[OPEN-ILS-GENERAL] Bib records not indexed - Evergreen 1.6.0.3

[Dan Scott]

On Thu, 2010-09-09 at 08:04 +0200, Linda Jansova wrote:
> 
> It seems that the use of letters such as "Ú" or "Č" at the beginning
> of 
> fields such as 100 a or 710 a may cause the problem (although it is 
> merely an assumption of ours).
> 
> Is there any way how this complication can be overcome and the
> records 
> get indexed properly? 

In Evergreen 1.6.x, as part of the indexing, all records get passed from
Perl through a server-side JavaScript routine then back to Perl, and
that seems to be a little shaky with handling Unicode. We've found in
the past that some characters need their Unicode normalized to NFC
(composed), and some need to be normalized to NFD (decomposed), to pass
through the JavaScript routine safely, and some don't make it through at
all (http://markmail.org/search/?q=NFC+NFD+import
+list:org.georgialibraries.list.open-ils-dev). I suspect this is what is
causing your problems.

The good news is that Evergreen 2.0 does away with the server-side
JavaScript routine and performs all of the indexing directly in the
database, so it should "just work". I'll try importing some of those
records in a 2.0 test server when I get a chance, but I'm confident that
it will work. The bad news is that Evergreen 2.0 is still only at an
alpha release stage at this point, so relief won't be available in a
stable release for some time yet.