[OPEN-ILS-DEV] MARC Batch Editor usage brought down app server

[Jason Stephenson]

Quoting Josh Stompro <stomproj at exchange.larl.org>:

> (Moving this over to Dev list from general)
>
> I was looking through the logs a little more, and it looks like the  
> MARC batch editor is able to add bib ID's to a bucket at 274 a  
> second(just counting one random second of logs).  It took 25 seconds  
> to add my 6700 bib id's to the merge bucket.
>
> If I create an insert statement with those same 6700 entries it only  
> takes 91ms to complete.  Would it be possible to send a batch of  
> record IDs in one request?  Could a cstore json query be constructed  
> that would try to insert all the fields in one statement, or grouped  
> into 500 at a time?

Welcome to the joys of using OpenSRF for batch operations!

Your best bet is actually filling the bucket from a SQL statement.

Cstore JSON queries can't do insert, update, nor delete, but they can
call functions to do that, so if you added a database function to
populate your bucket you could call that.

Many of the "batch" operations in the open-ils.* calls end up calling
the non-batch operation in a loop with a single transaction wrapper,
so they really don't improve performance.

For best performance through open-ils calls, you'd want to add a
storage call that really does a batch operation in the database.

> I also noticed that each run, successful or not seems to leave  
> behind a bucket entry.  On my 3 runs that got cut short because of  
> me restarting services, all the item entries remain also.  Should  
> those ever be cleaned up?  Should  
> "open-ils.cat.container.template_overlay.background" remove the  
> bucket entry when it is all done?

I have no idea what that method does nor what it should do. As I said
above, I typically use SQL. I also make it a habit to delete the bucket
entries first. Buckets will happily store duplicate entries and since
the user interfaces filter those out, you won't notice until you do
something via the database with that bucket.

My hunch, though, is that call is applying some sort of template to
the objects pointed at in the bucket. If that is the case, I would not
expect it to clear the bucket when it is done. You may want to keep
the entries in the bucket to do something else with them later.

HtH,
Jason



-- 
Jason Stephenson
Assistant Director for Technology Services
Merrimack Valley Library Consortium
4 High ST, Suite 175
North Andover, MA 01845
Phone: 978-557-5891
Email: jstephenson at mvlc.org