[Evergreen-dev] marc_export - apache crashes

Mon Oct 16 08:54:59 EDT 2023

Just want to say thank you to people for helping us out with the marc export (especially Josh Stompro was really helpful in getting our commands right and Jason Stephenson, I believe, updated the export to make it work better and pointed me to some good resources).

We ran an export over the weekend that worked successfully without crashing the server. 1.4 million records, we did it in batches of 100k per file and it was roughly 40 min. or so per 100k records. Some of our bibs have A LOT of items attached, so I didn't think this was crazy in timing.

Thanks again - really appreciate the support of this community!
Brian

Brian Holda
Library Technology Manager
Hekman Library
Calvin University
(616) 526-8673

[cid:061f7729-20eb-4246-89ab-42e69fe89b31]<https://library.calvin.edu/>

________________________________
From: Brian Holda <brian.holda at calvin.edu>
Sent: Monday, October 9, 2023 8:56 AM
To: Evergreen Development Discussion List <evergreen-dev at list.evergreen-ils.org>; Brian Holda <brian.holda at calvin.edu>
Subject: Re: marc_export - apache crashes

Or maybe a better way to ask this. Have people found a good way to export a large number of marc records within Evergreen? We found the staff client way to do it. And it processes files of 5-10,000 records at a time. But if we want to do 1 million records, let's say, it's a bit tedious. So then I found the marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=>. But that crashed our server doing it with 3,000 records at a time. We have ideas on how to modify the process, and it's not terrible using the staff client way, but I figure this must be a somewhat common task that others have good solutions for? Anyone willing to share 🙂?

Thanks,
Brian

Brian Holda
Library Technology Manager
Hekman Library
Calvin University
(616) 526-8673

[cid:e83c175b-fa22-41ee-b0e4-dc4102c98d50]<https://library.calvin.edu/>

________________________________
From: Evergreen-dev <evergreen-dev-bounces at list.evergreen-ils.org> on behalf of Brian Holda via Evergreen-dev <evergreen-dev at list.evergreen-ils.org>
Sent: Thursday, October 5, 2023 4:25 PM
To: Evergreen Development Discussion List <evergreen-dev at list.evergreen-ils.org>
Subject: [Evergreen-dev] marc_export - apache crashes

Hi all,

Not sure if it's user error or something else going on, so wanted to see if any of you all have experience using marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=> and had similar problems.

In brief:

  *   Tue, 5pm - I ran the following test (this is for a file of 3,100 records). This took about 30 sec. and successfully created the export file without any noticeable effects on our apache2 server:
cat /home/opensrf/marc-test.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml     -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > exported_files.xml
  *
Wed, 11:40am- I ran what I thought was essentially the same test (for the same file of 3,100 records). This also took about 30 sec. and successfully created the export file. However, 8 min. later apache crashed and had to be restarted. In the error log, it said "couldn't grab the accept mutex" immediately before crashing. Here's the code I ran:
cat /tmp/marc-output/marc1.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml     -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /tmp/marc-output/exported-marc1.xml
  *
Wed, 4pm- I ran essentially the same command (for the same file of 3,100 records), but without using the tmp folder. This time it stalled and after waiting a few minutes we pressed ctrl + c which I assumed stopped everything cleanly, as it returned me to the command prompt. However, at 4:50pm apache quit again, with the same "couldn't grab the accept mutex" messages beforehand. Here's the code I ran this time:
cat /home/opensrf/marc2.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml \ -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /home/opensrf/exported-marc2.xml

Anyone know what might be happening here?

Brian Holda
Library Technology Manager
Hekman Library
Calvin University
(616) 526-8673

[cid:4e579103-9d9b-445c-a644-73d36151b8e7]<https://library.calvin.edu/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231016/accdf3b1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-0wpikkpm.png
Type: image/png
Size: 5783 bytes
Desc: Outlook-0wpikkpm.png
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231016/accdf3b1/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-lsptfh50.png
Type: image/png
Size: 5783 bytes
Desc: Outlook-lsptfh50.png
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231016/accdf3b1/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-430ib0i4.png
Type: image/png
Size: 5783 bytes
Desc: Outlook-430ib0i4.png
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231016/accdf3b1/attachment-0005.png>