[OPEN-ILS-GENERAL] Importing Bibliographic Information

[Don McMorris]

Hi Jason et al:

There was a report on -dev back in early June
(http://list.georgialibraries.org/pipermail/open-ils-dev/2007-June/001228.html).
 His records started swapping quite a bit around 50k/records.  From
what I understand, he had to reset the database and change the script
queries.  The above message and the few surrounding it may be
worthwhile in reading.

I believe this bug has been entered/recognized.  However, as the data
load is expected to be a 1-time operation for the life of the system,
it hasn't been given as high a priority as the day-to-day stuff...
AAMOF, this would be a good place for somebody to make some
contributions ;).

If you have any more questions about data load, you might find -dev
more helpful (http://libmail.georgialibraries.org/mailman/listinfo/open-ils-dev).
 Admittedly, most of the people there are also here ;).  However, the
-dev list has historically been used for Tech Support too (and I know
of at least 2 other data load threads that took place on there).

Cheers!
--Don

On 7/26/07, Jason Stephenson <jstephenson at mailserv.mvlc.lib.ma.us> wrote:
> Don, et al.,
>
> Thanks for the information about the 852 tag. This is very helpful.
>
> I have so far tried importing about 850,000 bib records into Evergreen
> from our Horizon database, and the process did not go well. Apparently,
> our Core 2 Duo with 4 GB of RAM cannot handle processing all 850,000
> bibs at once, so I will have to break it up into chunks.
>
> I'm wondering if you have any benchmarks for how much RAM/CPU resources
> the conversion programs use per X number of records. The
> direct_ingest.pl on my test machine segfaulted after about 250,000
> records. I'm assuming that I'll need to load them in batches of 50,000
> to 100,000 to be successful.
>
> Also, for Deanna, I have a specific question. Can you tell me what
> parameters you used for your export target in Horizon to get your item
> info into the 852, please? I'm looking at this for our consoritum right
> now and while I have some ideas, it would save me a lot of time teasing
> it out if I had someone else's to look at.
>
> Cheers,
> Jason
>
>
>