<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I have been thinking about this a bit more. I was knew there was a need to prevent stale confs. The time performance is something that I’m not able to test. My machine has an SSD, a fast 4 core CPU and gobs of RAM. So I need you to keep me in line. ;)<div class=""><br class=""></div><div class="">The easiest way to keep it pristine is to unpack it into a temporary folder, rename the old folder and then rename the new folder. Finally deleting the old folder. By doing it in this order it minimizes the time that mods.d is unavailable. Important for multi-threaded apps and multiple apps that share the same machine simultaneously.<br class=""><div class=""><br class=""></div><div class="">Right now the SwordBookMetaData remembers the File for the conf of installed modules and is able to re-read it easily. But it does not store anything about a conf’s location when it is from mods.d.tar.gz. I suppose I could have it remember the location of mods.d.tar.gz and the name of the conf entry and create a method to extract a that conf out of the compressed archive. This would need to be done for each module that the user requests info. To do this is quite expensive as it means inflating the file then iterating over the contents until the desired conf is found.</div><div class=""><br class=""></div><div class="">I think that it would be better to see how much time it adds to extract the files and store them on disk. The fluffing of them would only be when the user wants to browse a description of the module.</div><div class=""><br class=""></div><div class="">I’d like to modify sbmd.toOSIS to check if the sbmd is partial or full and if not full re-read the conf fully and then continue as before. I think that is how JSword is designed to retreive the conf for presentation to the end user. Does AndBible use that or some other mechanism to get what it wants for presentation?</div><div class=""><br class=""></div><div class="">I think I’ll add a “fluff” method to BookMetaData that will do this. This could be called to get it to fluff at another time.</div><div class=""><br class=""></div><div class="">DM</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 11, 2016, at 1:00 PM, Martin Denham <<a href="mailto:mjdenham@gmail.com" class="">mjdenham@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">My rough estimates have the total size of conf files in all repos at about 5Mb which is not too different to the size of a module like ESV so the impact should not be significant and it should not be a problem if this is required.<div class=""><br class=""></div><div class="">Other things to consider that come to mind i) would need to remove conf files no longer in mods.d.tar.gz or delete and re-extract everything after a refresh ii) Time taken to save files - loading the list is already slow.</div><div class=""><br class=""></div><div class="">I can't think of any major reason not to do as you describe.</div><div class=""><br class=""></div><div class="">However, would an easier approach be to find files in the zip a bit like <a href="http://stackoverflow.com/questions/11123528/finding-a-file-in-zipentry-java" class="">this</a>. Speed would not be an issue because it would only be done once or twice after fetching the list e.g. to view About or to actually download. The mod.conf file name/path could be saved in SBMD if required.</div><div class=""><br class=""></div><div class="">Martin<br class=""><div class=""><br class=""></div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 11 January 2016 at 01:39, DM Smith <span dir="ltr" class=""><<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">I’m trying to figure out how to reload a conf from a remote source (to go from a partial load to a full load). The problem is that the AbstractSwordInstaller sits over top of mods.d.tar.gz, which it does not unpack. Instead, it iterates over all the entries in that binary file and handles each entry (i.e. a conf) in core. It doesn’t hit the disk. I’m wondering whether it would be alright to unpack the file in the same folder? That would allow a SwordBookMetaData to reload the file. It would also mean that SwordBookMetaData would only need one means of reading a conf as it’d be a file and not a byte array.<div class=""><br class=""></div><div class="">It isn’t a problem with desktop or server apps, but it might be for AndBible.<span class="HOEnZb"><font color="#888888" class=""><br class=""><div class=""><br class=""></div></font></span><div class=""><span class="HOEnZb"><font color="#888888" class="">— DM</font></span><div class=""><div class="h5"><br class=""><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 10, 2016, at 3:31 PM, DM Smith <<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>> wrote:</div><br class=""><div class=""><div style="word-wrap:break-word" class="">The problem you encountered was 2 bugs:<div class="">When the module is not UTF-8 the remote repository’s conf is re-read, but the filter wasn’t passed.</div><div class="">Not intended, but IniSection required a filter, rather than saying a null filter meant everything passed.</div><div class=""><br class=""></div><div class="">I’ve checked in that fix. Still trying to make the memory less….</div><div class=""><br class=""></div><div class="">— DM</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 10, 2016, at 1:18 PM, DM Smith <<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>> wrote:</div><br class=""><div class=""><div style="word-wrap:break-word" class="">The “Partial load of conf file.’ was to load all of the things in a conf that the JSword engine needs to work with a module. I don’t know why the CrossWire repo is working for me but not for you. I’ll keep working on it today. The problem with the previous commit was fixed with the last commit. I wasn’t “adjusting” the module after loading to fill in things like BookDriver and BookCategory.<div class=""><br class=""></div><div class="">I’m wondering whether getting the list of Books from the installer creates a deep rather than a shallow copy of them.</div><div class=""><br class=""></div><div class="">Today I hope to make SwordBookMetaData even more lazy. It has a BookDriver and validates its storage when the repo is loaded. I plan to break one of my modules by renaming one of the files and see the impact. Chris and I have noticed that the FileState objects are not fully released. This actually is part of the design.</div><div class=""><br class=""></div><div class="">Anyway, I think it is going in the right direction. Reducing the memory 4x is a good thing. The data structures within the IniSection may be too heavy. I may relax the requirement that it maintains the SWORD confs order. The idea was to be able to modify the provided conf, retaining its order. However, now we never modify that conf.</div><div class=""><br class=""></div><div class="">configAll was a deep clone of configSword. configAll adds in the contents of configJSword and then configFrontend. These last two are created even if not needed. We could make them lazy as well.</div><div class=""><br class=""></div><div class="">DM</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 10, 2016, at 11:07 AM, Martin Denham <<a href="mailto:mjdenham@gmail.com" target="_blank" class="">mjdenham@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">Thanks for the quick response. I have had a brief look at the new commits.</div><div class=""><br class=""></div><div class="">A lot of the attributes aren't being returned now so it is tricky to test and there are various errors but running the current tip '<a href="https://github.com/crosswire/jsword/commit/80020f51c6a762d458ce8ae70007b78eadee1fb3" title="Partial load of conf file.
Fixed a bug from the last commit in the loading of mods.d.tar.gz, adjustConfig was accidentally deleted.
This is part 1. Part 2 will be adding the ability to reload fully." style="font-size:13px;color:rgb(64,120,192);outline:0px;font-family:Helvetica,arial,nimbussansl,liberationsans,freesans,clean,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';line-height:20px;white-space:nowrap;background-color:rgb(245,245,245)" target="_blank" class="">Partial load of conf file.</a>' the SBMD for eBible is now only a quarter of the original size at 10Mb which is fine but I still don't understand why it is so large for the minimal attribute set now being returned.</div><div class=""><br class=""></div><div class="">I get a lot of errors like:</div><div class=""><div class="">SwordBookMetaData(492): Book not supported: malformed conf file for [BBE] no ModDrv found.</div><div class="">SwordBookMetaData(492): Malformed conf file: missing [BBE]Description=. Using BBE</div></div><div class=""><br class=""></div><div class="">and peculiarly the eBible repo seems to be the only repo I can use because all the others error.</div><div class=""><br class=""></div><div class="">I also tried the previous commit <a href="https://github.com/crosswire/jsword/commit/cc32ba8f1bb245932a747390d03874b2be70e9a1" title="Cut the memory requirements of a SwordBookMetaData in half." style="color:rgb(78,87,91);outline:0px;font-family:Helvetica,arial,nimbussansl,liberationsans,freesans,clean,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:15px;font-weight:bold;line-height:21px;background-color:rgb(247,251,252)" target="_blank" class="">Cut the memory requirements of a SwordBookMetaData in half.</a> but it did not work because basic attributes like language were not being returned.</div><div class=""><br class=""></div><div class="">I still don't understand why removing configSword should reduce memory by half because it should just be removing references to data that is also referenced from configAll, so it would reduce memory slightly but not much.</div><div class=""><br class=""></div><div class="">Martin</div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 10 January 2016 at 04:14, DM Smith <span dir="ltr" class=""><<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">OK. That’s done. Also accidentally introduced a bug with the last commit. It is noticeably fast.<div class=""><br class=""></div><div class="">Next up, allow for *a* SwordBookMetaData to be reloaded fully. This is needed to bring in all the other elements which are information only, such as About, in order to display info to the end user. Since the user will only look at one modules info at a time, it will load that one. You may need to change your code (hope not) to force that one to reload.</div><div class=""><br class=""></div><div class="">Give the code a try to see if it solves your out of memory error.</div><span class=""><font color="#888888" class=""><div class=""><br class=""></div><div class="">DM</div><div class=""><br class=""></div></font></span><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class=""><div class="">On Jan 9, 2016, at 9:06 PM, DM Smith <<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>> wrote:</div><br class=""></div></div><div class=""><div class=""><div class=""><div style="word-wrap:break-word" class="">I’ll be adding a filter to IniSection. Something like:<div class="">if (filter.test(key)) {</div><div class=""><span style="white-space:pre-wrap" class=""> </span>use the key</div><div class="">} else {</div><div class=""><span style="white-space:pre-wrap" class=""> </span>do nothing</div><div class="">}</div><div class=""><br class=""></div><div class="">SwordBookMetaData will be responsible for building the filter. At least for a first go around. A single object should do.</div><div class=""><br class=""></div><div class="">DM</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jan 9, 2016, at 6:29 PM, DM Smith <<a href="mailto:dmsmith@crosswire.org" target="_blank" class="">dmsmith@crosswire.org</a>> wrote:</div><br class=""><div class=""><div style="word-wrap:break-word" class=""><div class=""><div class=""><blockquote type="cite" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div class=""><div dir="ltr" class=""><div class=""><br class="">Yes, like you I have thought of streamlining conf loading for repo lists. One idea I had was to enable specification of a filter to SwordBookMetaData to limit the conf values that are stored.</div></div></div></blockquote><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><br class=""></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">I was thinking of something similar. My ideas aren’t good enough to be put into practice, but some kind of flag indicating empty, partially or fully loaded. Empty would mean that it hasn’t gone to disk to get the conf. Partial means that it read everything, but threw away most as not interesting (since the conf does not have order you have to read and parse it all). Full would mean that nothing was pitched. SwordBookMetaData.getProperty would need to be changed to determine whether the key is in memory or might be on disk and do the right thing. Or we could keep getProperty as it is and if you want one of the fields that is not stored (e.g. About) you have to call reload().</span></div><div class=""><br class=""></div><div class="">Maybe we could also cache that info into a separate file(s)? When mods.d.tar.gz is updated then the cache would be recomputed. In doing the computation, each conf would be read then pitched. Basically, the storage would be o.c.c.utils.Ini, if one file or IniSection, if many files.</div><div class=""><br class=""></div><div class="">What do you think?</div></div></div><br class=""></div></blockquote></div><br class=""></div></div></div></div><span class="">_______________________________________________<br class="">jsword-devel mailing list<br class=""><a href="mailto:jsword-devel@crosswire.org" target="_blank" class="">jsword-devel@crosswire.org</a><br class=""><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class=""></span></div></blockquote></div><br class=""></div></div><br class="">_______________________________________________<br class="">
jsword-devel mailing list<br class="">
<a href="mailto:jsword-devel@crosswire.org" target="_blank" class="">jsword-devel@crosswire.org</a><br class="">
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" rel="noreferrer" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">jsword-devel mailing list<br class=""><a href="mailto:jsword-devel@crosswire.org" target="_blank" class="">jsword-devel@crosswire.org</a><br class=""><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class=""></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">jsword-devel mailing list<br class=""><a href="mailto:jsword-devel@crosswire.org" target="_blank" class="">jsword-devel@crosswire.org</a><br class=""><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class=""></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">jsword-devel mailing list<br class=""><a href="mailto:jsword-devel@crosswire.org" target="_blank" class="">jsword-devel@crosswire.org</a><br class=""><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class=""></div></blockquote></div><br class=""></div></div></div></div></div></div><br class="">_______________________________________________<br class="">
jsword-devel mailing list<br class="">
<a href="mailto:jsword-devel@crosswire.org" class="">jsword-devel@crosswire.org</a><br class="">
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" rel="noreferrer" target="_blank" class="">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">jsword-devel mailing list<br class=""><a href="mailto:jsword-devel@crosswire.org" class="">jsword-devel@crosswire.org</a><br class="">http://www.crosswire.org/mailman/listinfo/jsword-devel<br class=""></div></blockquote></div><br class=""></div></div></body></html>