[OPEN-ILS-DEV] Re: pg_loader getting killed

[John Fink]

Excellent, thank you!

jf

On Thu, Feb 21, 2008 at 9:49 AM, Dan Scott <denials at gmail.com> wrote:

>  pg_loader-output.sql - it will run all the other scripts
>
> On 21/02/2008, John Fink <john.fink at gmail.com> wrote:
> > Okay, it's finished up, but I've got a bit of a problem -- with
> > pg_loader.pl, I get one sql file, but after I've run the parallel
> pg_loader,
> > I get many many many.
> >
> > I'm following the Gutenberg MARC conversion instructions:
> >
> > perl parallel_pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or
> > msfe -or mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a
> > msefe < full-bibs.ingest > full-bibs.sql
> >
> > I end up with these files, instead of one.
> >
> > pg_loader-output.bre.sql   pg_loader-output.mkfe.sql   pg_loader-
> > output.msfe.sql
> > pg_loader-output.mafe.sql  pg_loader-output.mrd.sql    pg_loader-
> > output.mtfe.sql
> > pg_loader-output.mfr.sql   pg_loader-output.msefe.sql  pg_loader-
> output.sql
> >
> > So which one should I load?  Or am I supposed to load all of them?  If
> so,
> > does it matter what order I load them in?
> >
> > jf
> >
> > On Wed, Feb 20, 2008 at 11:24 AM, John Fink <john.fink at gmail.com> wrote:
> >
> > > Oh man, I love you guys.  Thanks!  I'll give it a spin and report.
> > >
> > > jf
> > >
> > >
> > > On Feb 20, 2008 11:15 AM, Mike Rylander <mrylander at gmail.com> wrote:
> > >
> > > > On Feb 20, 2008 10:26 AM, John Fink <john.fink at gmail.com> wrote:
> > > > > Oh, and this is Evergreen 1.2.1.0, sorry for not stating that
> earlier.
> > > > >
> > > >
> > > > Grab parallel_pg_loader.pl from trunk* and try that.  It avoids
> > > > loading the entire file into memory before writing the files, and
> was
> > > > built for just such a situation.  It works fine with 1.2.x
> > > >
> > > > *
> > > >
> >
> http://svn.open-ils.org/trac/ILS/browser/trunk/Open-ILS/src/extras/import/parallel_pg_loader.pl
> > > >
> > > > --miker
> > > >
> > > > > jf
> > > > >
> > > > >
> > > > >
> > > > > On Feb 20, 2008 9:18 AM, John Fink <john.fink at gmail.com> wrote:
> > > > >
> > > > > > Hey folks,
> > > > > >
> > > > > > I've tried a couple of times unsuccessfully to run pg_loader on
> what
> > > > is
> > > > > admittedly a vastly huge (ls shows me filesize 17077683247) ingest
> > > > file.
> > > > > For a while it hums merrily along, but then I inevitably get
> something
> > > > like
> > > > > the following:
> > > > > >
> > > > > > 8643000 363.398018751032Killed
> > > > > >
> > > > > > Complete with a zero-length sql.  Am I simply biting off more
> than I
> > > > can
> > > > > chew?  Is it possible to easily split the ingest file for easier
> > > > loading?
> > > > > It looks like to me that pg_loader is trying to load everything
> into
> > > > memory
> > > > > before writing the sql (hence the zero length).
> > > > > >
> > > > > > jf
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > http://libgrunt.blogspot.com -- library culture and technology.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > http://libgrunt.blogspot.com -- library culture and technology.
> > > >
> > > >
> > > >
> > > > --
> > > > Mike Rylander
> > > >  | VP, Research and Design
> > > >  | Equinox Software, Inc. / The Evergreen Experts
> > > >  | phone:  1-877-OPEN-ILS (673-6457)
> > > >  | email:  miker at esilibrary.com
> > > >  | web:  http://www.esilibrary.com
> > > >
> > >
> > >
> > >
> > > --
> > > http://libgrunt.blogspot.com -- library culture and technology.
> > >
> >
> >
> >
> > --
> > http://libgrunt.blogspot.com -- library culture and technology.
> >
>
>
> --
> Dan Scott
> Laurentian University
>



-- 
http://libgrunt.blogspot.com -- library culture and technology.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20080221/2b60439a/attachment-0001.html