[OPEN-ILS-GENERAL] Running authority_control_fields.pl

[Jeff Davis]

On Fri, 2 May 2014 16:50:25 +0000, Kyle Tomita <kTomita at catalystitservices.com> wrote:
> I recently ran the auth to bib linking with the use of the batcher
> scripts from MVLC,
> http://blog.mvlcstaff.org/2012/09/howto-batch-authority-control.html.
> 
> It took about 8 days and 8 hours to complete.
> 800K bib records and 560K auth records
> Ran batches of 5000 bibs and ran 3 batches at a time.
> 
> I am wondering if others have used this batching process and how my
> times match up.

Last time I tried running authority_control_fields.pl, it took about 30
minutes to process a set of 1,554 bib records on a test server, i.e.
about 1.2s per record.  Since it was just a small test set, I didn't try
running multiple processes in parallel.  I didn't notice particularly
high load during the test, so I suspect disk I/O was the limiting
factor; I'd expect better results on a db server using SSDs, but haven't
tested that so far.
-- 
Jeff Davis
Lead Evergreen Specialist
BC Libraries Cooperative