[OPEN-ILS-GENERAL] Importing holdings data

[Anoop Atre]

Hi John, What system are you leaving? We moved a library from Unicorn 
last year and have code we can share. It might be useful to look through 
in any case. Your plan looks pretty good, I didn't write the code but my 
co-worker did something similar, we found running scripts in parallel 
cut processing time quite a bit.

http://svn.mnpals.net/viewvc/evergreen/egdemo/local_unicorn_conversion/

Cheers
-- 
- - - - - - - - - - - - - - - - - - - - - - - - -
Anoop Atre
IS Developer & Integrator, MnPALS
PH: 507.389.5060
OF: 3022 Memorial Library (Office-ML 3022)
-- 
"Mit der Dummheit kämpfen Götter selbst vergebens"
  ~ Johann Christoph Friedrich von Schiller


On 01/14/2011 12:02 AM, John Morris wrote:
> On Thu, 2011-01-13 at 21:42 -0500, Dan Scott wrote:
>> On 13 January 2011 21:13, John Morris<jmorris at beau.org>  wrote:
>>> On Thu, 2011-01-13 at 20:36 -0500, Jason Etheridge wrote:
>>>
>>>>> fingerprint: Eh?  So how would I generate this?
>>>>
>>>> There should be a trigger to handle this for you (maybe depending on
>>>> your version of EG).  It's used for grouping records with similar
>>>> titles into metarecords (think FRBR-ish, but for lumping different
>>>> formats and editions together).  See the metabib.metarecord table.
>>>
>>> Nope, don't see any trigger functions on the biblio table.  So for now
>>> assuming it either isn't important or whatever needs it will generate
>>> it.  Otherwise it can be fixed later.
>>
>> John - this suggests that you're using 1.6, in which ingest (indexing
>> and all of that goodness) happens outside of the database.
>
> Yup, 1.6.0.8 for now.  Seemed the best option, the upgrade page for
> 1.6.1 is pretty spartan and posts here on this list imply 1.6.2 was a
> big mistake that should not have escaped into the wild while 2.0 isn't
> ready for production yet.  So should I just move to 2.0 anyway?
>
>> This is why you run the MARC records through marc2bre.pl to generate
>> the BRE (biblio.record_entry) JSON objects; then you run those BRE
>> JSON objects through direct_ingest.pl to generate all of the JSON
>> objects representing indexes, fingerprints, etc. Then you run it
>> through (parallel_)pg_loader.pl to generate the SQL that you can load
>> into the database. If you don't run the direct_ingest.pl step, then
>> you're not going to be able to search for your records (well, you'll
>> be able to search for them, but you won't find any hits).
>
> I though the indexing was done in quick_metarecord_map.sql.  If it isn't
> I'm pretty much boned.  One option I see is that the record number can
> be forced to be taken from a field in the record so I could rig that.
> Then when processing holdings out of those same records I'd know the
> record id.
>
>> In 2.0, you can skip the direct_ingest.pl step because the database
>> automatically generates all of the indexes, fingerprints, etc as a
>> result of triggers.
>>
>>> Thanks for filling in the other details.  Since in theory I now have all
>>> of the information needed all that remains is writing the code
>>> Hopefully that won't take very long.  Will be starting it tonight.
>>
>> Good luck! Hopefully you have copy locations, circulation modifiers,
>> copy status, open circulation transactions, etc all covered too. If
>> you only have one copy location and very simple circulation rules then
>> those might not apply to you.
>
> Copy location and status are already in the exported holdings records.
> The others I have plans for,  Basically my migration plan involved these
> steps, mostly to be implemented in one big script that connects to both
> database backends and starts tossing records across while reformatting
> them.
>
> 1.  Using a processed MARC dump instead of our current db I was to get
> bib and holdings.  Original idea was to just use the batch import
> feature in Evergreen itself but that was not viable because it can't
> read it's output format back and it isn't even clear that it can be
> coaxed into completely processing a bib+holding without any manual help.
> At any rate there is insufficient documentation.
>
> 2.  Connect to both databases and bring across the users.  This part is
> working.
>
> 3.  After those are in bring across current circulations, hold requests
> and outstanding fines by carefully translating the internal ids of each
> system and a few other twiddles.
>
> 4.  Time permitting bring across old circulation, acquisitions, anything
> else interesting.  But none of those are required for the initial
> cutover since they can be backfilled before decommissioning the old
> system.
>
> 5.  Win.  That plan isn't totally viable anymore but something close to
> it might be salvagable.
>


-- 
- - - - - - - - - - - - - - - - - - - - - - - - -
Anoop Atre
IS Developer & Integrator, MnPALS
PH: 507.389.5060
OF: 3022 Memorial Library (Office-ML 3022)
-- 
"Mit der Dummheit kämpfen Götter selbst vergebens"
  ~ Johann Christoph Friedrich von Schiller