[OPEN-ILS-DEV] Data load challenges

[Don Hamilton]

Thanks Mike.
 
 
 I added a couple of lines to pg-loader (to print "BEGIN" / "COMMIT"
statements) and removed the 'ticker'. My line 86 is:
 
        my $fields = join(',', @{ $fieldcache{$h}{fields} });
 
Do I have a set of records that is missing subject data? I do have the
author data in the sql file, followed by the .\. I DON'T have the copy
for the COPY for metalib.subject_field_entry.
 
AH... writing things down helps... I just reran the original 802
records, starting with record id 803, and that did not fail... so I
think the problem may be the that there are no subject fields in the
oxford data... which of the (many) parameters to pg-loader might I
remove to NOT try to load subjects....
 
don




>>> mrylander at gmail.com 7/5/2007 11:00 AM >>>

On 7/5/07, Don Hamilton <dhamilton at wlu.ca> wrote:
[snip]
>
> Then I try:
>
> perl marc2bre.pl -startid=803 ebooks/oxford.mrc |  perl
direct_ingest.pl |
> perl pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or msfe
-or
> mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a msefe
>
> oxford.sql
>
> and get:
>
> Writing file ...
> Can't use an undefined value as an ARRAY reference at pg_loader.pl
line 86,
> <> line 34657.

hmm... line 86 generates an SQL command to wipe the table being worked
on (based on a command line switch, which you didn't give), and I
don't see any arrayrefs there.  I suspect perl is actually complaining
about the line above that, though I see no reason for that either,
unless the IDL is broken (which it doesn't seem to be, at least not
for mafe or msfe (the next one)).

I'm assuming that the command you pasted above was copied directly
from the shell command line.  Have you modified you're copies of
pg_loader.pl or fm_IDL.xml by any chance?
>
>
> There's actually data in oxford.sql, with appropriate looking record
> numbers. The file ends with data for "COPY
metabib.author_field_entry
> (field,source,value) FROM STDIN;".
>
> So what should come after, and what am I missing?
>

You should be seeing lots (thousands) of lines for
metabib.author_field_entry, then \., then

COPY metabib.subject_field_entry(field,source,value) FROM STDIN;

and so on.

As for what we're missing, I'm not sure yet ... I'm trying to come up
with a relatively simple way to trim this down to the smallest
possible test case, assuming that something in the MARC is triggering
the failure, but pg_loader doesn't know anything about MARC -- it just
sorts and prints piles and piles of array-based objects as PG COPY
commands.

--miker

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20070705/a39a01e7/attachment.html