[OPEN-ILS-DEV] Updating a large set of bib records

[BUNTON, GLENN]

I would also be very interested in hearing successful ways of importing a large set of records into Evergreen. We would like to import close to 6 million records but from our limited knowledge and experience with the system it seems this might take forever. What is the relationship between the speed of importing records into Evergreen and hardware limitations. Are there configuration tweaks we can make to speed up the process, such as modifications of postgresql? Does the ejabberd settings impact record import in any manner?  Any insights will be greatly appreciated.


From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Kyle Tomita
Sent: Thursday, April 24, 2014 6:12 PM
To: open-ils-dev at list.georgialibraries.org
Subject: [OPEN-ILS-DEV] Updating a large set of bib records

Hi Everyone,

I am tasked with importing a large set of bibliographic marc records (under 1 million).

I have leveraged work from Jason Stephenson, http://git.mvlcstaff.org/?p=jason/backstage.git;a=summary.

The import script has been modified to instead of doing the update, to create sql files with the update commands.  These files have about 10,000 records per file.  This bypasses checking with database and just creates the update scripts off of the marc records.

These files are then batch processed.

This process ignores overlay profiles, which was deemed not needed for this process.

Before the update, triggers on the biblio.record_entry are turned off, particularly the reingest.  We run a full reingest after all the records have been updated.

My reason for posting this is to get feedback from others who are charged with updating a large set of bib records (over 500,000) about the way in which they succeeded and also pitfalls.

Kyle Tomita
Developer II, Catalyst IT Services
Beaverton Office
[cid:image001.png at 01CF6009.7C3DC9B0]



Sent from my Verizon Wireless 4G LTE smartphone

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20140425/1a0a67bf/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 8323 bytes
Desc: image001.png
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20140425/1a0a67bf/attachment-0001.png>