<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Well, hypothetically, we might be able to make a reasonable
attempt to teach the filter when to strip by determine which
adjacent character an accent might be modifying and conditionally
strip or not strip, but pragmatically, this filter is used to
remove Greek accents while searching Greek texts, enabling us to
perform accent-insensitive searching of the base material. I
suppose it might be useful, if a text interspersed occasional
Greek, to allow accent-insensitive searching of any of those
interspersed Greek words, but without a compelling use case now,
I'm not too inclined to improve the filter to be that much more
logical. If someone else sees a compelling need, by all means add
the extra heuristic logic. I don't consider this a bug right
now. The filter is meant to be used on UTF-8 Greek text. I
certainly receive the suggestion it might be used in another case,
as you suggest, and would consider that an improvement, if we ever
have a solid use case.</p>
<p>Troy<br>
</p>
<br>
<div class="moz-cite-prefix">On 02/21/2017 03:12 PM, DM Smith wrote:<br>
</div>
<blockquote
cite="mid:1DA16A68-D818-4E60-9B8E-3E70A7CF5959@crosswire.org"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hypothetical: What about mixed language texts such as a
Greek/French lexicon?
<div class=""><br class="">
</div>
<div class="">DM</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 21, 2017, at 4:56 PM, Troy A. Griffitts
<<a moz-do-not-send="true"
href="mailto:scribe@crosswire.org" class="">scribe@crosswire.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class=""><br class="">
Simply don't use the UTF-8 Greek Accent filter on
non-Greek texts. As you have discovered there are
accents used in Greek which are also used in other
languages and adverse effects will be seen for these
languages. The bottom line is simple. Only use the UTF-8
Greek Accents filter on UTF-8 Greek texts.<br class="">
<br class="">
Hope this helps.<br class="">
<br class="">
<div class="gmail_quote">On February 21, 2017 2:45:24 PM
MST, David Haslam <<a moz-do-not-send="true"
href="mailto:dfhmch@googlemail.com" class="">dfhmch@googlemail.com</a>>
wrote:
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<pre class="k9mail">These are the principal diacritics found in Biblical Greek that have to be
removed with a UTF8GreekAccents filter.
The first five are general accents, not particular to Greek.
It's on account of these that the filter should not be applied to non-Greek
text.
U+0300 ̀ COMBINING GRAVE ACCENT
U+0301 ́ COMBINING ACUTE ACCENT
U+0308 ̈ COMBINING DIAERESIS
U+0313 ̓ COMBINING COMMA ABOVE
U+0314 ̔ COMBINING REVERSED COMMA ABOVE
U+0342 ͂ COMBINING GREEK PERISPOMENI
U+0343 ̓ COMBINING GREEK KORONIS
U+0344 ̈́ COMBINING GREEK DIALYTIKA TONOS
U+0345 ͅ COMBINING GREEK YPOGEGRAMMENI
No other diacritics or characters should be removed.
Though there are a few more combining accents in this block, they aren't
really used in Biblical Greek.
I am open to correction on this point.
e.g. The right single quotation mark (U+2019) is NOT a diacritic. It should
not be removed.
Before any of these accents can be removed, they must first be separated
from the Greek letters they are combined with.
Although normalization to the decomposed form can produce this effect, as we
have seen already, this can have undesirable side effects on any non-Greek
text in the module that may happen to include combined or unusual
characters.
It would therefore be more sensible to simply use a comprehensive mapping
table that replaces each possible accented character by the corresponding
letter in the Greek alphabet. In this way the filter can completely avoid
the need to apply any Unicode normalization.
The complete mapping table would have at least 130 rows. It will need to
take into account that there are at least 75 possible combinations of a
letter with two accents. There are none with three.
Any residual combining characters should also be removed, to cover the
possibility that a module may have been intentionally made without
normalizing the Greek source text by default to NFC.
That's my proposal. I can easily create such a mapping table that
programmers can use.
I can also readily test it with a bespoke TextPipe filter.
Best regards,
David
--
View this message in context: <a moz-do-not-send="true" href="http://sword-dev.350566.n4.nabble.com/GlobalOptionFilter-UTF8GreekAccents-and-non-Greek-modules-tp4656719p4656765.html" class="">http://sword-dev.350566.n4.nabble.com/GlobalOptionFilter-UTF8GreekAccents-and-non-Greek-modules-tp4656719p4656765.html</a>
Sent from the SWORD Dev mailing list archive at <a moz-do-not-send="true" href="http://nabble.com/" class="">Nabble.com</a>.
<hr class="">
sword-devel mailing list: <a moz-do-not-send="true" href="mailto:sword-devel@crosswire.org" class="">sword-devel@crosswire.org</a>
<a moz-do-not-send="true" href="http://www.crosswire.org/mailman/listinfo/sword-devel" class="">http://www.crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre></blockquote></div>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.</div>_______________________________________________
sword-devel mailing list: <a moz-do-not-send="true" href="mailto:sword-devel@crosswire.org" class="">sword-devel@crosswire.org</a>
<a moz-do-not-send="true" href="http://www.crosswire.org/mailman/listinfo/sword-devel" class="">http://www.crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</div></blockquote></div>
</div>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
sword-devel mailing list: <a class="moz-txt-link-abbreviated" href="mailto:sword-devel@crosswire.org">sword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://www.crosswire.org/mailman/listinfo/sword-devel">http://www.crosswire.org/mailman/listinfo/sword-devel</a>
Instructions to unsubscribe/change your settings at above page</pre>
</blockquote>
</body></html>