[OPEN-ILS-DEV] Data load challenges

[Mike Rylander]

On 7/5/07, Don Hamilton <dhamilton at wlu.ca> wrote:
[snip]
>
> Then I try:
>
> perl marc2bre.pl -startid=803 ebooks/oxford.mrc |  perl direct_ingest.pl |
> perl pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or msfe -or
> mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a msefe >
> oxford.sql
>
> and get:
>
> Writing file ...
> Can't use an undefined value as an ARRAY reference at pg_loader.pl line 86,
> <> line 34657.

hmm... line 86 generates an SQL command to wipe the table being worked
on (based on a command line switch, which you didn't give), and I
don't see any arrayrefs there.  I suspect perl is actually complaining
about the line above that, though I see no reason for that either,
unless the IDL is broken (which it doesn't seem to be, at least not
for mafe or msfe (the next one)).

I'm assuming that the command you pasted above was copied directly
from the shell command line.  Have you modified you're copies of
pg_loader.pl or fm_IDL.xml by any chance?

>
>
> There's actually data in oxford.sql, with appropriate looking record
> numbers. The file ends with data for "COPY metabib.author_field_entry
> (field,source,value) FROM STDIN;".
>
> So what should come after, and what am I missing?
>

You should be seeing lots (thousands) of lines for
metabib.author_field_entry, then \., then

COPY metabib.subject_field_entry(field,source,value) FROM STDIN;

and so on.

As for what we're missing, I'm not sure yet ... I'm trying to come up
with a relatively simple way to trim this down to the smallest
possible test case, assuming that something in the MARC is triggering
the failure, but pg_loader doesn't know anything about MARC -- it just
sorts and prints piles and piles of array-based objects as PG COPY
commands.

--miker