[OPEN-ILS-GENERAL] Importing holdings data

[John Morris]

On Thu, 2011-01-13 at 21:42 -0500, Dan Scott wrote:
> On 13 January 2011 21:13, John Morris <jmorris at beau.org> wrote:
> > On Thu, 2011-01-13 at 20:36 -0500, Jason Etheridge wrote:
> >
> >> > fingerprint: Eh?  So how would I generate this?
> >>
> >> There should be a trigger to handle this for you (maybe depending on
> >> your version of EG).  It's used for grouping records with similar
> >> titles into metarecords (think FRBR-ish, but for lumping different
> >> formats and editions together).  See the metabib.metarecord table.
> >
> > Nope, don't see any trigger functions on the biblio table.  So for now
> > assuming it either isn't important or whatever needs it will generate
> > it.  Otherwise it can be fixed later.
> 
> John - this suggests that you're using 1.6, in which ingest (indexing
> and all of that goodness) happens outside of the database.

Yup, 1.6.0.8 for now.  Seemed the best option, the upgrade page for
1.6.1 is pretty spartan and posts here on this list imply 1.6.2 was a
big mistake that should not have escaped into the wild while 2.0 isn't
ready for production yet.  So should I just move to 2.0 anyway?

> This is why you run the MARC records through marc2bre.pl to generate
> the BRE (biblio.record_entry) JSON objects; then you run those BRE
> JSON objects through direct_ingest.pl to generate all of the JSON
> objects representing indexes, fingerprints, etc. Then you run it
> through (parallel_)pg_loader.pl to generate the SQL that you can load
> into the database. If you don't run the direct_ingest.pl step, then
> you're not going to be able to search for your records (well, you'll
> be able to search for them, but you won't find any hits).

I though the indexing was done in quick_metarecord_map.sql.  If it isn't
I'm pretty much boned.  One option I see is that the record number can
be forced to be taken from a field in the record so I could rig that.
Then when processing holdings out of those same records I'd know the
record id.

> In 2.0, you can skip the direct_ingest.pl step because the database
> automatically generates all of the indexes, fingerprints, etc as a
> result of triggers.
> 
> > Thanks for filling in the other details.  Since in theory I now have all
> > of the information needed all that remains is writing the code
> > Hopefully that won't take very long.  Will be starting it tonight.
> 
> Good luck! Hopefully you have copy locations, circulation modifiers,
> copy status, open circulation transactions, etc all covered too. If
> you only have one copy location and very simple circulation rules then
> those might not apply to you.

Copy location and status are already in the exported holdings records.
The others I have plans for,  Basically my migration plan involved these
steps, mostly to be implemented in one big script that connects to both
database backends and starts tossing records across while reformatting
them.

1.  Using a processed MARC dump instead of our current db I was to get
bib and holdings.  Original idea was to just use the batch import
feature in Evergreen itself but that was not viable because it can't
read it's output format back and it isn't even clear that it can be
coaxed into completely processing a bib+holding without any manual help.
At any rate there is insufficient documentation.

2.  Connect to both databases and bring across the users.  This part is
working.

3.  After those are in bring across current circulations, hold requests
and outstanding fines by carefully translating the internal ids of each
system and a few other twiddles.

4.  Time permitting bring across old circulation, acquisitions, anything
else interesting.  But none of those are required for the initial
cutover since they can be backfilled before decommissioning the old
system.

5.  Win.  That plan isn't totally viable anymore but something close to
it might be salvagable.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://libmail.georgialibraries.org/pipermail/open-ils-general/attachments/20110114/62bf39fc/attachment-0001.pgp