[sword-devel] indexed search discrepancy

[Matthew Talbert]

The problem is more universal and serious than I originally thought.
SWORD indexed search is performing rather poorly against BT's. At any
rate, the biggest issue is the size given to MAX_CONV_SIZE in
swmodule.cpp. Here are some tests:

//default value of 2047
./search Finney "good" | wc
[0=================================50===============================100]
 ======================================================================
     22     255    1549


//MAX_CONV_SIZE = 10000
./search Finney "good" | wc
[0=================================50===============================100]
 ======================================================================
     51     576    3573

//MAX_CONV_SIZE = 15000
./search Finney "good" | wc
[0=================================50===============================100]
 ======================================================================
     56     650    3985

But even 15000 isn't high enough to get all occurrences of words at
the end of long text sections. For Finney, a value of 20000 is
probably required and it's entirely possible that other modules would
require higher values.

I don't know what the consequences are of changing this value, but
currently we're missing a huge number of hits in genbook modules and a
substantial number of hits in commentaries as well.

If this gets fixed, I think searching and results should be added to
the test suite. It would be simple to add; just run mkfastmod, then
the search program (it would be nice to be able to change the search
type without re-compiling so that different search types could be
done).

Matthew