[OPEN-ILS-DEV] Problem Importing MARC records

[Don Hamilton]

Hi Karen.

I have done some playing with the load process. I think you failed to
set a 'start record' number in marc2bre.... once you have any records in
the database, you need to determine the next start number, and pass it
to marc2bre... for example,

 perl marc2bre.pl -startid=2017 ebooks/gale.mrc |  perl
direct_ingest.pl -idler 0 |  perl pg_loader.pl -or bre -or mrd -or mfr
-or mtfe -or mafe -or msfe -or mkfe -or msefe -a mrd -a mfr -a mtfe -a
mafe -a msfe -a mkfe -a msefe -idler 0 > gale.sql

Hmm... That reminds me... in order to make the output of such piped
commands legible, I modified direct_ingest and pg-loader to not put out
the record count/records per second stuff that each program normally
does, hence the "-idler 0" in the above command. I guess I'd better get
my head back in this space long enough to submit a patch... With
existing code, use something like:


 perl marc2bre.pl -startid=2017 ebooks/gale.mrc |  perl
direct_ingest.pl  |  perl pg_loader.pl -or bre -or mrd -or mfr -or mtfe
-or mafe -or msfe -or mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a
msfe -a mkfe -a msefe > gale.sql

don (not mcmorris, and not dan, either)


>>> kcollier at kent.lib.md.us 02-Aug-2007 12:02 PM >>>
I went through the .marc file and found the records containing some of
the strange characters, and went back and looked at those records in
Horizon.  They display as diamond shapes with question marks there, and
the odd capitalization you mentioned shows up there as well.  Our
cataloger is looking at those records, intending to correct them.  

Meanwhile, I tried using a smaller set of MARC records that didn't
include any of the records I had problems with in the .marc to .bre
phase.  Everything went smoothly with no apparent errors until I got
to:

psql -U evergreen evergreen
evergreen# \i ~/kent1000records.sql;

and I got a whole series of these guys again:





----- Original Message -----
From: Joe Atzberger <atzberger at MECDC.ORG>
To: open-ils-dev at list.georgialibraries.org 
Sent: Wed,  1 Aug 2007 17:24:36 -0400
Subject: Re: [OPEN-ILS-DEV] Problem Importing MARC records


> Karen --
> 
> Is there something strange going on with your character set (and
> capitalizations)?  In your records, what are the 
> characters that display in the error log (or at least in my email) as
squares?
> Perhaps they are illegal escape 
> characters or something more appropriate in UTF-8.
> 
> Is is possible your line endings were reformatted or scripts randomly
forced to
> uppercase?  To me, "nEW yORK" looks like 
> the result of a tr/[A-z]/a-Z/ and "\N" looks suspiciously like the
newline
> character "\n".
> 
> --Joe Atzberger
> INFOhio Technical Specialist
> Metropolitan Educational Council
> (614) 473-8300 x22
> atzberger at mecdc.org 
> 
> 
>