[OPEN-ILS-DEV] import data from voyager

[Dan Scott]

On 24/05/07, Jason Zou <qzou at lakeheadu.ca> wrote:
> Don Hamilton wrote:
> > Congratulations, Jason... You beat me by a long shot.... But now
> > that you have, (and now that Rene has our system starting with NO
> > logged errors!) can you give us some details of what exactly you did?
> > Once I build on what you've done and loaded some records, I'll
> > volunteer for a wiki account and add a 'voyager data load' section
> > once we're done.
> >
> > Also, could you define "very slow"? As I mentioned in a query a week
> > or so ago, I'd like to iterate through loading some 5 million bibs,
> > and if 10,000 are very slow, I'm afraid to think what 5m would be. I
> > have some experience with bulk loading to fresh data bases, and will
> > take a stab at that if the 'regular' load is too slow.
> >
> > don
> >
> > >>> qzou at lakeheadu.ca 5/24/2007 10:44 AM >>>
> > Hi everyone,
> >
> > By using scripts in the Open-ILS/src/extra/import/, I imported about
> > 11,000 MARC records. Although the process was very slow, it seems that
> > it is working. And records have been added into the following tables:
> >           biblio.record_entry
> >           metabib.rec_descriptor
> >           metabib.full_rec
> >           metabib.title_field_entry
> >           metabib.author_field_entry
> >           metabib.subject_field_entry
> >           metabib.keyword_field_entry
> >           metabib.series_field_entry
> >
> > But when I tried to use OPAC to find some records, I always got nothing.
> > I am wondering whether there are something that I missed out.
> >
> > Any suggestions are highly appreciated.
> >
> > Jason
> >
> > Lakehead University
> >
> Hi Don,
>
> Thanks. I just want to see how fast the importing will be. I exported
> 12,343 records from our Voyager database. It took me about 1.5 hours to
> load them into Evergreen.
> In comparison with the converting process, surprisingly, it only took
> one minute or less to store records into Pg database.
>
> By the way, my server is on a P4 2.4 GHz machine with 1.5G RAM running FC5.
>
> Jason
>
> Lakehead University
>
>

Strange. Here's the `time` results of running an import of the 14,449
Gutenberg records (including all of the steps described in my previous
email in this thread) on the Gentoo VMWare image with 512MB of RAM:

real    51m7.623s
user    39m24.824s
sys     2m58.639s

I'm not sure how my virtual machine with 1/3 of your physical
machine's RAM could possibly outperform your physical machine.
Something seems weird there. But I concur that the bulk of the time is
spent in the marc2bre.pl / direct_ingest.pl / pg_loader.pl processes.

-- 
Dan Scott
Laurentian University