[sword-devel] lucene indexing failing on some modules
dmsmith555 at yahoo.com
Fri May 9 13:57:08 MST 2008
Karl Kleinpaste wrote:
> I have found that indexing is not working on some modules, including
> some of my home-grown ones, but also a few Crosswire-distributed ones.
> Case in point, BosworthToller. This was first noticed from integrated
> support in GS, but use of mkfastmod on its own shows similar problems.
> In 15min wall clock time, mkfastmod grew to 126M and used 13:20 CPU
> without completing, or generating anything in the lucene subdir in the
> module's tree.
The code is in .../modules/swmodule.cpp.
The way indexing works is that:
It creates the location that the index should write to.
Then it uses a RAMDirectory to write the index to.
Then there is an ugly loop that iterates over the entries in a module,
getting their keys and contents, inverting them into the index.
Perhaps of interest is that the first MAX_CONV_SIZE (i.e. 2047) of the
content is used.
Once it is done, it opens an index on disk and writes the RAMDirectory
It would be a simple matter to add a diagnostic to the loop to have it
spit out the key.
Just a thought:
I would think that if one could run mod2imp successfully on the module
that the indexer would work.
More information about the sword-devel