[OPEN-ILS-DEV] Regarding Marc::Charset

[Mike Rylander]

On 6/21/07, Don Hamilton <dhamilton at wlu.ca> wrote:
> Hi Guys.
>
> I'm revisting an issue I looked at briefly a couple of weeks ago.
> Marc::Charset is spewing out lots of "no mapping found for [0x??] at
> position ?? in ???sometext??? / g0=ASCII_DEFAULT g1=EXTENDED_LATIN at
> /usr/local/share/perl/5.8.8/MARC/Charset.pm line 210."
> messages.
>
> I THINK this is because I already have Unicode Marc records (based on my
> last Endeavor/ExLibris/Voyager upgrade) so I don't really want, or need, to
> use Charset at all. I tried several variations in marc2bre of setting $enc
> to 'utf8', and ignore errors to 0 or 1, but the conversion errors just keep
> on a comin'...
>
> How might I stop this from happening?

If your records are truly UTF-8 encoded (and properly marked as such)
then you'll never see those errors.  So, assuming that the records are
UTF-8 encoded, the problem is that the leader is incorrectly marked as
MARC-8.

So, lets see if we can convert your records to MARCXML with
marcdumper, and then do the import dance (it should be much faster for
the marc2bre part).

In the current SVN trunk you'll find mardumper in

 Open-ILS/src/extras/marcdumper/

This is a modified version of IndexData's utility.  We've extended it
to accept a -r paramter, which when outputing XML allows you to strip
out data using XPath.

You'll need to run `make` in that directory to compile the binary.
Once you have that, use the following command to convert your MARC21
to MARCXML:

  ./marcdumper -f UTF-8 -t UTF-8 -X /your/marc21/file.mrc > /tmp/marc-output.xml


This will force the leader into the correct state.  Now you can give
/tmp/marc-output.xml to marc2bre, along with --marctype=XML, and your
records should come in fine.  If, however, you find your records
bombing out either during the marc2bre or ingest phases, you'll
probably want to run marcdumper again with a -f of MARC8.

Let us know how that goes.

TIA,

--miker