[OPEN-ILS-DEV] marc_export and Perl Version >= 5.20

[Jason Stephenson]

Hi, all.

NOTE: This is not https://bugs.launchpad.net/evergreen/+bug/1671845. It
may be related or have a similar cause, but the experience/symptoms are
completely different.

At this point, consider this a head's up, as well as a problem
description that I don't yet think I have enough information to file as
a bug report. It is also a request for anyone who wants to double check
this report and to help with debugging.

I've noticed some bizarre behavior with DBI, MARC::Record, and writing
to a file with Perl version 5.20 and 5.22. (These versions of Perl ship
with Debian 8 Jessie and Ubuntu 16.04 Xenial Xerus, respectively.)

I have a script (https://github.com/Dyrcona/boopsie) that I use to make
a weekly extract of records to send to Boopsie, Inc. on behalf of our
member libraries that use their app.

What I have seen is that when run with the aforementioned versions of
Perl, the program consumes all of the RAM on the server and gets killed
by OOM killer. No output ever reaches the file. This suggests to me that
the problem occurs in the main loop with extract and converting the
MARCXML from the database, though it could be the Perl output buffer run
amok.

The main loop of my program is similar, though less complicated, than
that of marc_export. I tried marc_export to see if it would have the
same problem. When extracting my whole database, it does:

marc_export -a -e UTF-8 > all.mrc

It also crashes if fed the output of an equivalent psql query to extract
all of the record ids, or if a file of all record ids is piped into
marc_export. It makes no difference if the output format is USMARC or
MARCXML.

I can split this up into batches of 50,000 or so records (quite possibly
more) and all is well. I figured this out by dumping records for a
branch with around 51,000 items and that worked. My whole database has
just over 2.7 million, non-deleted bib records.

This worked on Perl version 5.14 on Debian 7 Wheezy and on Ubuntu 14.04
Trusty Tahr.

I hope to run marc_export with the Perl debugger to figure out the exact
cause. Until this is fixed, I'm using a work around in my scripts of
dumping MARCXML batches and converting them to USMARC and putting them
into 1 file with yaz-marcdump. This seems to work in light of the Lp bug
mentioned in the NOTE.

Any and all information, contradictory or otherwise, from those using
Debian 8 or Ubuntu 16.04 is most welcome.

Jason