Robinson's morphology is not indexed in JSword modules

See pull request = Doesn't look right on github, but I'm pretty sure I selected the branch. Perhaps it's because it's branched off something that's ahead of the jsword...

This has been pulled&merged: https://github.com/crosswire/jsword/pull/11 & https://github.com/crosswire/jsword/pull/10

Now in the build. see OSISUtil.getMorphologiesWithStrong() and LuceneIndex.FIELD_MORPHOLOGY

The entry now is something like G2345@robinson:T-NPN

There should not be robinson: in it.

While I like the functionality of G2345@T-NPN, we need to think about this. Do we want to have both strong: and morph: ? Will just one suffice?

If we have just one do we allow strong: and morph: searches? Or do we have a new name, e.g. lex: ?

I think we do want to have both. One allows for exact matches on a strong number (strong: ) which is presumably faster than a trailing wildcard search. The morphology will most likely be using wildcards most of the times (e.g. all adjectives for this word).

So the question, do we not want to have "robinson:" in the indexed field? If codes clash (unlikely I know), we wouldn't want to return the wrong results for some modules.

The clash is less unlikely as soon as you introduce wildcards, e.g. G1000@ * A * (ignore spaces - JIRA changed to bold otherwise) or whatever the code is going to be. A could mean adverb.

If we are going to change it, then what is your suggestion? morphField.substring(morphField.indexOf(':') + 1)

We don't want robinson: any more than we want strong: w/in the other field.
Same with strongMorph: used in the OT.

BTW, the whole use of the morph field is not very useful. The least significant value of the field is in the most significant position. Wildcard searches won't be very helpful.

I think this (how to make searching useful) is a good topic for sword-devel.

I disagree in terms of wildcards. You need to use wildcards to search for all adjectives, or all plurals.

It's a perfectly good thing to do:

• search for all verbs in the passage (regardless of their strong numbers)
• or search for all the times a particular word is used as a verb

I use the morph field quite successfully in STEP (although in the browser, rather than in a search), to highlight plurals in bold, singular normal font. And then I highlight masculines and feminines in blue / red. I use regular expressions, but a carefully crafted query using wildcards could equally do the job.

The first of the two points, you'd obviously be looking for terms rather than verses. The second is easy enough to do looking for verses and using a custom highlighter or something like that. But they key thing here, is STEP will have a feature where we allow the user to choose a word (by strong number effectively) and then select which grammar points he wants to search across. Then give him the resulting verses, highlighting the strong number.

So a user may want mercy, used as a noun and in the singular mode.

Really? How would you search for adverbs using wildcards? Wouldn't that also pull back adjectives?

How do you find plural verbs? How about first person voice (this one is tough as it is the unmentioned default)?

How do you find all uses of a word as a verb in aorist form?

Robinson codes are unnecessarily difficult to search, and it would be possible to design something easier for computer searches, but they are consistent in their complexity and can be searched by RegEx.
The key to them is to distinguish the following types of words:

ALL X - Tense Voice Mood - Person Case Number Gender (- Extra)
Participles X - Tense Voice Mood - Case Number Gender (- Extra)
Other Verbs X - Tense Voice Mood - Person Number (- Extra)
1+2 Pronouns X - Person Case Number (- Extra)
Refl+Poss Pn X - Person Case Number Gender (- Extra)
Other Pronouns X - Case Number Gender (- Extra)
Art, Nouns + Adj X - Case Number Gender (- Extra)

(Uggg - the layout of this is lost - so line up the columns in your head)

In RegEx:
Verbs [V]-[2]*[PIFARLX][AMPEDONQX][ISOMNPR]-[123][NADG][SP][MFN][\w-]
Others [V]-[123]*[NADG][SP][MFN][\w-]
CONJ, PRT etc [A-Z][A-Z][A-Z][\w-]*

For searches this can be summarised as:

1) Participles
ie any that start with "V-" and include "P-"
2) Other verbs
ie any that start with "V-" but no "P-"
3) Nouns, articles, adjectives and general pronouns
ie starting "N-", "A-", "P-" followed by a letter, "D-"
3) Person Pronouns (1st+2nd+Reflex.+Poss)
ie like nouns but add Person (ie a number) after the first hyphen (and sometimes have Gender at the end)
4) Unparsed words
ie ADVerbs, PARTicles, PREPositions - ie anything that doesn't have a hyphen as the second character

Rules for searches:

Tense is the 1st letter after "V-"
Voice is the 2nd letter after "V-"
Mood is the 3rd letter after "V-"
Person is the a number after "^V-"
Case is the 1st letter after "^V-"
Number is any S or P after "^V-"
Gender is the 3rd letter after "^V-"
(where "^V" is any letter except V)

RegEx examples:

• Aorist tense = V-[2]*A[\w-]*
  all tenses = V-[2]*[PIFARLX][\w-]*
• Active voice = V-[2]*[A-Z]A[\w-]*
  all voices = V-[2]*[A-Z][AMPEDONQX][\w-]*
• iNfinitive mood = V-[2]*[A-Z][A-Z]N[\w-]*
  all moods = V-[2]*[A-Z][A-Z][ISOMNPR][\w-]*
• Second person = [\w-]*[^V]-2[\w-]*
  all persons = [\w-]*[^V]-[123][\w-]*
• Nominative case = [\w-]*[^V]-\d*N[\w-]*
  all cases = [\w-]*[^V]-\d*[NADG][\w-]*
• Plural number = [\w-]*[^V]-\w*P[\w-]*
  all numbers = [\w-]*[^V]-\w*[SP][\w-]*
• Neuter gender = [\w-]*[^V]-\d*[A-Z][A-Z]N[\w-]*
  all genders = [\w-]*[^V]-\d*[A-Z][A-Z][MFN][\w-]*

Looking at the code now. There's something about it that isn't quite right.

In the OT we don't have morphology codes that have semantic meaning. In fact, the codes (strongsMorph aka "thayers") that are used are placeholders for the future. We don't have a dictionary for those codes. I'd love it if David I-B, Daniel Owens and Chris Little would work together to come up with a meaningful code pattern that can be used. Chris has talked about a code that would not require an external dictionary. As long as we have a mapping from existing (robinson and "thayer") codes to the new one, I'll do the work in the KJV to update it.

But regarding the code, we should put the morph codes in to the searchable morph field regardless if they are Robinsons or strongsMorph.

The form of SN@MC (Strong's Number@Morph Code) is useful if the Morph Code has semantic meaning (as above).

The other thing, is that the lemma, morph and src attribute values form parallel arrays. Here are 3 examples from the upcoming KJV of the longest ones:
<w src="2 3 4 5" lemma="strong:G3956 strong:G3739 strong:G5100 strong:G302 tr:παν tr:ο tr:τι tr:αν" morph="robinson:A-ASN robinson:R-ASN robinson:X-ASN robinson:PRT">whatsoever</w>
<w src="15 16 17 18" lemma="strong:G3588 strong:G4286 strong:G3588 strong:G740 tr:η tr:προθεσις tr:των tr:αρτων" morph="robinson:T-NSF robinson:N-NSF robinson:T-GPM robinson:N-GPM">the shewbread</w>
<w src="10 13 14 15" lemma="strong:G3756 strong:G1519 strong:G3588 strong:G165 tr:ουκ tr:εις tr:τον tr:αιωνα" morph="robinson:PRT-N robinson:PREP robinson:T-ASM robinson:N-ASM">never</w>

It is not a valid assumption that they always form parallel arrays. It took a lot of effort to clean up the KJV NT to make it so. Here is an example from the KJV OT:
<w morph="strongMorph:TH8799 strongMorph:TH8675 strongMorph:TH8686" lemma="strong:H05749 strong:H05749">What thing shall I take to witness</w>

There's no reason to think other modules are error free. See an example from ABP here:
http://www.crosswire.org/tracker/browse/MOD-245
This has Strong's Numbers but they are pretty goofy.

So I think we should do the following:
Have one or more fields with SN, MC and SN@MC.

The construction of SN@MC needs to be best effort, pairing the first Strong's Number with the first morph code, the second with the second and so forth until no more pairs can be formed.

The question I have is do we really need more than one field if the codes are not overlapping?

And I wonder about the - in the code. I was under the impression that the - formed a word break. It seems to me that we need to normalize the code so that it can be searched well.

Similar question regarding @ does it get stripped out by Lucene? Do we need to protect it?

I'm inclined to say that the easiest path is 3 fields: strong:SN, morph:MC and lex:SN@MC.

There's a few different issues here:

A) Greek OT:

1) The ABP problems with OT Greek codes (examples at MOD-245) are not easy to fix, because this is a copyright text.
2) The problems with OT Greek vocabulary stem from the fact that the OT has words which aren't in Strong's lexicon, cos this was made for the NT. Ideally the lexicon should be extended in a way that is backwardly compatible - perhaps like the NASB does for Hebrew.
3) There is no OT Greek morphology in the ABP module (or any other OSIS module as far as I know).

STEP has permission to use the CCAT data for Greek OT lexicon and morphology (I'm not sure if we needed to ask, but we did). Implementing this is not straightforward, but it will happen.

B) Hebrew OT:

1) The Hebrew morphology codes are fairly straightforward, but not full.
They only tell us the Stem and Mood for verbs, and supply no information about nouns, particles etc.
2) There are some additional codes marked as morphology which are actually notes such as "the Qere is followed here instead of the Ketiv")
3) The KJV module, as coded at present, cannot be used to create arrays, because there are frequently more lemmas than morphology codes in the same tag, with no indication as to which morphology goes with which lemma. This is due to problems (1) and (2).
4) About 6000 words in the OT have no lemma tags.

These issues are now all fixed - the work was done by the Open Scripture guys and myself - and I'm in the process of checking it.
It would be great to indegrate this into the KJV module.

Hebrew Morphology:

The Open Scripture guys have have a very well thought-out system of morphology codes and are creating a crowd-sourcing tool to get this implemented. However, this is probably a very long-term project.

I am in the process of adding a much more limited set of morphology codes in the mean-time, ie Stem, Mood, person, number, gender - for all words.
The codes will not be perfect, but they will supply something while the Open Scripture morphology is being developed.

Question:

Given that the Robinson codes are 'standard', do you think the OT Hebrew codes should try to emulate them, or should we start again with something which is easier to search and process?

I was thinking of something where every word would have the same morphology layout,
eg: VQi3SF meaning Verb: Qal Imperative 3rd singular feminine
or: N---PM meaning Noun: plural masculine

The idea is that each morphology would have standard place markers for Stem, Mood, person, number, gender
so (for example) plurals would always have "P" as the 5th character.

I'm not sure I'm the best to answer the OT Hebrew Codes question. I took a 7 week crash course in Hebrew, but I don't remember enough of it to weigh in. I'd defer to Chris L, our resident linguist, or to you or some other Hebrew scholar.

But I'll give an end user kind of answer:
The pattern should serve two goals well. The casual user who needs to decode it as a glance and the advanced user who will form search requests.

For someone that has passing knowledge of Greek, the Robinson codes are fairly easy to learn and quick to read. As we allow the user to show/hide these values, this is important. I'd want the same for people with a working knowledge of Hebrew.

If I were to pick, I would not like to count to see that the P is in the 5-th character and that for nouns, I have to see 3 dashes. A dash as a separator works fine visually, but not as a placeholder. Some fonts will show it as a single long dash. So I'd suggest a different order, such that their would be as few dashes as possible. No need to have trailing dashes. And perhaps a / or another character that visually will never join an adjacent character and will not need special processing when showing in HTML/XML.

The other part is that the end user will perhaps be forming search queries manually. In lucene, the don't care matcher for a single character is a dot (or maybe it is a question mark. I forget.) So the user would type 'N...PM' to search. Having a different order would give NPM* or perhaps just NPM, if we don't right pad the code.

But that's just a first response, I might find that I quickly adjust to 3 dashes in the middle.

One of the basic human user interface principles is to order the material from the most useful to the least useful, from left to right, from top down. I think such would apply to this as well (though not top down).

Great idea about changing the order.

Nouns have number, gender
Particles usually have nothing
verbs can have everything
Adjectives may have gender and number
Adverbs usually have nothing
Object marker have number and perhaps gender
Pronouns have number, gender & person

They all have a Type (ie 'Noun','verb' etc)
And they all have a Language (Hebrew, Aramaic or personal Name)

So to minimise place markers the order would be:
Language Type, Number, Gender Person, and then verbal details.

For verbs, we could simply record Stem (eg Qal, Hiphil etc)
but there are 37 of these in Strong's system, and more in others,
and it is more meaningful to interpret the Stems as Voice and Mood

So, we end up with:
Language, Type, Number, Gender, Person, Voice, Mood, State/Aspect

eg
HNSM = Hebrew Noun Singular Masculine
Aa = Aramaic adverb
NLSF = Name of Location Singular Feminine
HVPM2PIC = Hebrew Verb Plural Masculine 2nd person Passive Intensive Consecutive imperfective

I've attached a table of all the abbreviations ("Encoding.doc")

Regarding using regular expressions to do a search:
Lucene search syntax is not regular expression. It is more like unix command-line globbing. I haven't seen regular expression support in a contrib to Lucene, but that doesn't mean it is not there.

But if not, to support regular expressions, we'll need to intercept the query and pick out the regular expression and use the regular expression to do our own search over our own store or the term dictionary.

The RegEx expressions were more complicated than I had thought they would be.
Is it time to redesign the Robinson Codes?
They aren't particularly human-friendly or machine-friendly
I think the latter is more important because ideally people won't see the actual coding.

Agreed - showing the codes to the user, should be a last resort thing, as it implies that they need to learn the new system.

It seems Lucene has some support for Regular Expressions anyway: http://lucene.apache.org/core/old_versioned_docs/versions/3_0_3/api/contrib-regex/org/apache/lucene/search/regex/package-summary.html

If we can create a mapping for Robinson codes to something that is better (human readable and easy to search), then we can use the mapping w/in JSword to provide a better user experience.

Basic thought, the user would see the new codes or a decoding of these codes into their language (or the default, if there's no such translation). They can search these codes either directly or via a wizard (what is done would be a front-end choice).

It may be that the underlying module uses the old codes. That'd be ok. Not ideal. The search would reverse the mapping going from the new codes to the old codes and use that to search the module. Likewise, when presenting the module, the old codes would be replace with the new codes. This would be a process of normalization, which we do currently for Strong's numbers.

We may want to explore the idea of a module sidecar. On various occasions, I've wanted finer grain information regarding a module. Basically, we'd maintain a separate conf for the modules. It'd contain information regarding thing like: user provided font info, unlock keys, type of Strong's numbers per testament, type of morphology per testament, .... Any program can set a value into the sidecar. This info would be read into BookMetadata and would be available for all programs. If a program doesn't know what to do with it, it'd ignore it. It would be good to communicate and document these new values. Automatic behavior that's added to JSword would need to be discussed.