[sword-devel] linking / testsuite
jmarsden at fastmail.fm
Wed Apr 22 18:21:51 MST 2009
Ben Morgan wrote:
> If you have 66 books of the Bible and many different ways of
> referencing them, testing every single way isn't necessarily all that
> easy. Then multiply that by every locale that is there.
A regression test doesn't (usually) try to give complete coverage of
anything, just enough tests to convince the team that functionality that
is now "correct" does not later become "incorrect" without someone
quickly noticing the change.
In this case, testing (say) 3 common abbreviations for each book name
followed by the first verse of the first chapter, for each of the 66
books, is 66 * 3 = 198 tests, and just using a single very boring locale
(either C or en_US.UTF-8) would catch what happened recently -- and so
would notify developers by failing if the change were to accidentally
disappear (the wrong revision is tagged/checked out of svn by mistake,
say) between now and the official 1.6 release, for example.
In practice, a really "good" regression test for abbreviations might
include a couple of other locales too, perhaps one Western European one
that needs a few accented Latin characters (French or German or Spanish,
say), and one that needs its own non-Roman alphabet (Cyrillic, or
Korean, or Arabic, maybe).
At this level of coverage, we'd be looking at a maximum of 66 * 3 * 3 =
under 600 test items. That's entirely doable, surely?
Beyond that, subject matter experts for each locale might come up with
"special" "known to be peculiar" abbreviations they feel it would be
good to add as additional tests... but that's moving away from the realm
of basic regression testing, really, I think.
If I have time later tonight, once I have a sword 1.6.0RC2 package out
for front end developers to test with, I'll see about creating the 198
item single locale test and (if I succeed!) I'll contribute it to the
SWORD codebase, just to demonstrate that creating such a test really is
in fact doable :)
> I think the vk1 = vk2 change is
> more important to test than the other, really.
You provided no basis for this opinion. I think that *any* significant
change that happens after an rc release is worthy of a regression test,
more or less by definition. My reasoning: the developers thought the
codebase was good enough to release, and it turned out not to be. This
most likely means that the bugs found were unexpected or subtle or hard
to test for, and yet were deemed significant enough to warrant fixing
before final release. Therefore, surely, each such bug deserves a test
in the regression test suite.
Basically, bugs you find in an RC release are often the "creme de la
creme" of bugs... treat them with respect :)
More information about the sword-devel