[OPEN-ILS-DEV] MARC Batch Editor usage brought down app server

[Josh Stompro]

(Moving this over to Dev list from general)

I was looking through the logs a little more, and it looks like the MARC batch editor is able to add bib ID's to a bucket at 274 a second(just counting one random second of logs).  It took 25 seconds to add my 6700 bib id's to the merge bucket.

If I create an insert statement with those same 6700 entries it only takes 91ms to complete.  Would it be possible to send a batch of record IDs in one request?  Could a cstore json query be constructed that would try to insert all the fields in one statement, or grouped into 500 at a time?
I also noticed that each run, successful or not seems to leave behind a bucket entry.  On my 3 runs that got cut short because of me restarting services, all the item entries remain also.  Should those ever be cleaned up?  Should "open-ils.cat.container.template_overlay.background" remove the bucket entry when it is all done?

Josh

From: Open-ils-general [mailto:open-ils-general-bounces at list.georgialibraries.org] On Behalf Of Josh Stompro
Sent: Friday, February 12, 2016 1:28 PM
To: open-ils-general at list.georgialibraries.org
Subject: [OPEN-ILS-GENERAL] MARC Batch Editor usage brought down app server

I was attempting to use the MARC batch editor to add an 856 field to 6000 records last night and managed to take out one of our application servers.  It stopped responding to requests and wouldn't allow users to log on or perform functions if they were already logged on.

I think what I managed to do was to start 3 different batch edit sessions.  I was using the load from CSV method to select records.  I tried twice with the 6000 record file, and both times I received an internal server error message.  Then I tried to cut down the file to 1000 records, and tried it again.  And it seemed like it attached to one of the previous attempts because it all of a sudden started at 1300 records and gave me progress updates up until 4300 or so, and then the client seemed to lock up on me.

At that point I started getting after hours phone calls from our largest branch about Evergreen no longer working.  After I restarted the opensrf services and apache the system started working again.

It looks like open-ils.auth is what failed on me and took out normal functionality for that node.  When restarting services, I saw a timeout on shutting down auth, it had to be forced to terminate.

I tried to find an error in the logs that explained better what happened, but I wasn't able to find anything specific, which is probably more just not knowing what to look for.

I wonder if I managed to DOS the server.  While looking through the logs I saw that what the MARC Batch Editor does is loads the CSV file data into a record bucket, one record at a time.  Then it goes through and does an update record permission check before submitting the batch edit request.  Do the permission checks use open-ils.auth? I see that there were 7 auth drones when the system became unresponsive, but our config says that up to 15 can run.

What are a reasonable number of records to process at a time using the MARC Batch Editor?
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20160213/9fa2fd7f/attachment.html>