[OPEN-ILS-GENERAL] Importing records from Unicorn

[Dan Scott]

2008/10/8 Warren Layton <warren.layton at gmail.com>:
> On Tue, Oct 7, 2008 at 12:30 AM, Dan Scott <denials at gmail.com> wrote:
>> 2008/10/6 Warren Layton <warren.layton at gmail.com>:
>>> * YAZ version 2.1.56
>>> * MARC::Record is from the Ubuntu package "libmarc-record-perl", version 2.0.0-1
>>> * marc2bre I'm less sure. I got it from the import demo on the Evergreen site
>>> (http://open-ils.org/~denials/import_demo.tar.gz)
>>
>> Hmm - that could be a problem. Assuming you're running against
>> Evergreen trunk, you will probably want to use the versions of
>> marc2bre.pl, parallel_pg_loader.pl, direct_ingest.pl and
>> quick_metarecord_map.sql from trunk. I ran the import_demo against
>> 1.2.3.1 last week and noticed that the namespaces in the generated XML
>> didn't quite look right in comparison to records loaded via Vandelay.
>> import_demo was last updated seven months ago, so it probably needs an
>> update to sync up with what Evergreen expects today.
>>
>> You can also upgrade yaz to 3.0.36; that's what is used in the most
>> current version of Makefile.install.
>
> Update: I upgraded yaz and grabbed the latest marc2bre.pl and this
> seems to work for the first steps.
>
> Once I get to the execution of parallel_pg_loader.pl, I see a few
> warnings like this:
>   "Argument "\\N" isn't numeric in int at parallel_pg_loader.pl line
> 110, <> line 2263351."
>
> A more serious problem occurs when the loader finishes and I try to do
> a "\i pg_loader-output.sql" on the psql prompt:
>
> evergreen=# \i /home/opensrf/import_demo/pg_loader-output.sql
> SET
> BEGIN
> psql:/home/opensrf/import_demo/pg_loader-output.sql:5: ERROR:
> duplicate key violates unique constraint "record_entry_pkey"
> CONTEXT:  COPY record_entry, line 186269: "t    now     1       f
>  now     1       canadianentomologistentomological       10010488
>  IMPORT-1223388363.688   <record    xmlns:xs..."
> psql:/home/opensrf/import_demo/pg_loader-output.sql:6: ERROR:  current
> transaction is aborted, commands ignored until end of transaction
> block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:7: ERROR:  current
> transaction is aborted, commands ignored until end of transaction
> block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:8: ERROR:  current
> transaction is aborted, commands ignored until end of transaction
> block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:9: ERROR:  current
> transaction is aborted, commands ignored until end of transaction
> block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:10: ERROR:
> current transaction is aborted, commands ignored until end of
> transaction block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:12: ERROR:
> current transaction is aborted, commands ignored until end of
> transaction block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:13: ERROR:
> current transaction is aborted, commands ignored until end of
> transaction block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:14: ERROR:
> current transaction is aborted, commands ignored until end of
> transaction block
> psql:/home/opensrf/import_demo/pg_loader-output.sql:15: ERROR:
> current transaction is aborted, commands ignored until end of
> transaction block
> evergreen=#
>
>
> When running marc2bre.pl, I used the option "--idfield=001". My guess
> was that the first error in the above block indicates that there are
> two identical 001 fields among the records. Is this the correct
> interpretation of this error?

That sounds right to me. You're working with Unicorn, right? They
don't force unique 001s. They even have multiple 001s in the same
record.

The trick with Unicorn is that you need to ensure that you've exported
the unique catalog key in the bibs and holdings and use that for your
ID field. You also want to ensure that you "include local fields" -
most sites have 856 configured as "local" for example, and without
that you get no URLs.

Unfortunately, I don't think the delivered reports for Unicorn include
the option to export the unique catalog key (at least, not in the Java
WorkFlows GL3.1 interface that I'm looking at). Does anyone at your
site have Unicorn API training, or can you bring in somebody with
Unicorn API access? If so, the Unicorn export script that I posted in
the API repository should help a lot.

-- 
Dan Scott
Laurentian University