[OPEN-ILS-DEV] JSON Character Encoding Problem (direct_ingest.pl)

[Dan Wells]

Hello,

I am getting the following error on just a handful of records when running direct_ingest.pl:

Couldn't process record: invalid character encountered while parsing JSON string, at character offset 115 ["\x{8}derschaft],","q..."] at /usr/local/share/perl/5.10.0/OpenSRF/Utils/JSON.pm line 126.

I have isolated and attached the offending MARC records.  I get the same result whether I allow the import scripts to handle the UTF8 conversion or whether I handle it myself in an external utility.

I also found the following IRC log discussing this issue:

http://www.open-ils.org/irc_logs/openils-evergreen/2008-08/%23openils-evergreen.06-Wed-2008.log
(see starting at 2008-08-06T21:22:01)

Based on what I read here, I think I have found some opposite cases where NFD fails and NFC works.  Switching the entityize call in marc2bre.pl from 'D' to 'C' allows these records to be processed.

Any thoughts on this?  Are these actually bugs in JSON::XS?

Thanks,
DW

-------------- next part --------------
A non-text attachment was scrubbed...
Name: bad_rec_sample.mrc
Type: application/mrc
Size: 12375 bytes
Desc: not available
Url : http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20090512/26f12c00/attachment.bin