[OPEN-ILS-DEV] Conversion conundrums

[Don Hamilton]

OK, so, just for the record: (with a question at the end)
 
For now I just added hardcoded variables to marc2bre so I could
experiment a bit...
 
50,000 marc records seems to be the problem point in the process. Up to
30,000, and the pg_loader writes quickly just as soon as it has read the
input. When I try for 50,000, there's a long, swappy session before the
process ends.
 
Now, my question for you all, dear readers...
 
Is there something other than a "make all" that will reset the
bibliographic parts of the evergreen database to a pristine state? 
 
don

>>> dhamilton at wlu.ca 6/6/2007 9:40 AM >>>

Hi again.
 
Today, I'd like some advice/suggestions for loading huge gobs of marc
records.
 
In an attempt to use both halves of my dual cpu server, and do it all
at once, I tried 
 
"marc2bre | direct_ingest.pl | pg_loader.pl > sql"...
 
Two things are not right with this approach...
 
1) the three idlers overwrite each other
2) after about 3 hours of merrily churning away, I get "out of memory",
and the system has a schizophrenic swapping party until I kill the three
tasks.
 
I think I'd like to have marc2bre read two 'extra' parameters,
startatwhat and howmanyshouldIdo, and well, start at the startat record
in the input file and do howmanyshouldido...
 
BUT..
 
Should I make them parameters? Or should they be read/written from a
control source somewhere so the above command line can re-iterate itself
over a large input file?
 
don

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20070607/0df18bcf/attachment.html