[Evergreen-dev] marc_export - apache crashes

[Brian Holda]

Or maybe a better way to ask this. Have people found a good way to export a large number of marc records within Evergreen? We found the staff client way to do it. And it processes files of 5-10,000 records at a time. But if we want to do 1 million records, let's say, it's a bit tedious. So then I found the marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=>. But that crashed our server doing it with 3,000 records at a time. We have ideas on how to modify the process, and it's not terrible using the staff client way, but I figure this must be a somewhat common task that others have good solutions for? Anyone willing to share 🙂?

Thanks,
Brian


Brian Holda
Library Technology Manager
Hekman Library
Calvin University
(616) 526-8673

[cid:e83c175b-fa22-41ee-b0e4-dc4102c98d50]<https://library.calvin.edu/>



________________________________
From: Evergreen-dev <evergreen-dev-bounces at list.evergreen-ils.org> on behalf of Brian Holda via Evergreen-dev <evergreen-dev at list.evergreen-ils.org>
Sent: Thursday, October 5, 2023 4:25 PM
To: Evergreen Development Discussion List <evergreen-dev at list.evergreen-ils.org>
Subject: [Evergreen-dev] marc_export - apache crashes

Hi all,

Not sure if it's user error or something else going on, so wanted to see if any of you all have experience using marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=> and had similar problems.

In brief:

  *   Tue, 5pm - I ran the following test (this is for a file of 3,100 records). This took about 30 sec. and successfully created the export file without any noticeable effects on our apache2 server:
cat /home/opensrf/marc-test.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml     -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > exported_files.xml
  *
Wed, 11:40am- I ran what I thought was essentially the same test (for the same file of 3,100 records). This also took about 30 sec. and successfully created the export file. However, 8 min. later apache crashed and had to be restarted. In the error log, it said "couldn't grab the accept mutex" immediately before crashing. Here's the code I ran:
cat /tmp/marc-output/marc1.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml     -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /tmp/marc-output/exported-marc1.xml
  *
Wed, 4pm- I ran essentially the same command (for the same file of 3,100 records), but without using the tmp​ folder. This time it stalled and after waiting a few minutes we pressed ctrl​ + c​ which I assumed stopped everything cleanly, as it returned me to the command prompt. However, at 4:50pm apache quit again, with the same "couldn't grab the accept mutex" messages beforehand. Here's the code I ran this time:
cat /home/opensrf/marc2.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml \ -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /home/opensrf/exported-marc2.xml

Anyone know what might be happening here?


Brian Holda
Library Technology Manager
Hekman Library
Calvin University
(616) 526-8673

[cid:4e579103-9d9b-445c-a644-73d36151b8e7]<https://library.calvin.edu/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231009/a4c97a69/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-0wpikkpm.png
Type: image/png
Size: 5783 bytes
Desc: Outlook-0wpikkpm.png
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231009/a4c97a69/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-lsptfh50.png
Type: image/png
Size: 5783 bytes
Desc: Outlook-lsptfh50.png
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20231009/a4c97a69/attachment-0003.png>