[OPEN-ILS-GENERAL] MARC Batch Editor usage brought down app server

[Josh Stompro]

I was attempting to use the MARC batch editor to add an 856 field to 6000 records last night and managed to take out one of our application servers.  It stopped responding to requests and wouldn't allow users to log on or perform functions if they were already logged on.

I think what I managed to do was to start 3 different batch edit sessions.  I was using the load from CSV method to select records.  I tried twice with the 6000 record file, and both times I received an internal server error message.  Then I tried to cut down the file to 1000 records, and tried it again.  And it seemed like it attached to one of the previous attempts because it all of a sudden started at 1300 records and gave me progress updates up until 4300 or so, and then the client seemed to lock up on me.

At that point I started getting after hours phone calls from our largest branch about Evergreen no longer working.  After I restarted the opensrf services and apache the system started working again.

It looks like open-ils.auth is what failed on me and took out normal functionality for that node.  When restarting services, I saw a timeout on shutting down auth, it had to be forced to terminate.

I tried to find an error in the logs that explained better what happened, but I wasn't able to find anything specific, which is probably more just not knowing what to look for.

I wonder if I managed to DOS the server.  While looking through the logs I saw that what the MARC Batch Editor does is loads the CSV file data into a record bucket, one record at a time.  Then it goes through and does an update record permission check before submitting the batch edit request.  Do the permission checks use open-ils.auth? I see that there were 7 auth drones when the system became unresponsive, but our config says that up to 15 can run.

What are a reasonable number of records to process at a time using the MARC Batch Editor?
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-general/attachments/20160212/ccf0a190/attachment.html>