[OPEN-ILS-GENERAL] Bib records not indexed - Evergreen 1.6.0.3

[Linda Jansova]

  Dan, thanks for the explanation and also for the positive news that 
Evergreen 2.0 should be able to handle these characters without problems 
:-)!

Deanna, I believe that USMARC or SUTRS syntax should not be the problem 
as other bib records in USMARC format have been processed correctly.

BTW, we encountered problems with these characters also in the older 
versions of Evergreen we experimented with earlier - an example (which 
is still "valid")  would be a record from the Library of Congress (TCN 
3760924 - or it can be found using Author Čapek, Karel and Title RUR 
:-). Maybe this could also be an example worth trying in Evergreen 2.0...

Linda

Dne 9.9.2010 17:08, Dan Scott napsal(a):
> On Thu, 2010-09-09 at 08:04 +0200, Linda Jansova wrote:
>> It seems that the use of letters such as "Ú" or "Č" at the beginning
>> of
>> fields such as 100 a or 710 a may cause the problem (although it is
>> merely an assumption of ours).
>>
>> Is there any way how this complication can be overcome and the
>> records
>> get indexed properly?
> In Evergreen 1.6.x, as part of the indexing, all records get passed from
> Perl through a server-side JavaScript routine then back to Perl, and
> that seems to be a little shaky with handling Unicode. We've found in
> the past that some characters need their Unicode normalized to NFC
> (composed), and some need to be normalized to NFD (decomposed), to pass
> through the JavaScript routine safely, and some don't make it through at
> all (http://markmail.org/search/?q=NFC+NFD+import
> +list:org.georgialibraries.list.open-ils-dev). I suspect this is what is
> causing your problems.
>
> The good news is that Evergreen 2.0 does away with the server-side
> JavaScript routine and performs all of the indexing directly in the
> database, so it should "just work". I'll try importing some of those
> records in a 2.0 test server when I get a chance, but I'm confident that
> it will work. The bad news is that Evergreen 2.0 is still only at an
> alpha release stage at this point, so relief won't be available in a
> stable release for some time yet.
>
>