[OPEN-ILS-DEV] marc_export and Perl Version >= 5.20

[Jason Stephenson]

Even more investigation reveals that I was jumping to conclusions.

Turns out that on a Debian 7 Wheezy machine with Perl 5.14, a simple
script to dump the MARC for all of our 2.7 million, non-deleted bib
records uses almost 12GB of RAM. Looks like this was always an issue,
and I was just running the scripts on hardware with more RAM.

I should still build a VM with a more recent Debian or Ubuntu release
with enough RAM for comparison.

Sorry for all of the noise.

On 04/05/2017 10:30 AM, Jason Stephenson wrote:
> More investigation points the finger at Perl DBI and/or DBD::Pg. They
> are now apparently caching all of the results before returning anything.
> My reading of the documentation seems to imply that this was always the
> case, however I was able to dump 2.7 million records with Perl 5.14 on a
> server with 8GB of RAM without running out of memory. With Perl 5.18+
> this appears to no longer be possible, YMMV.
> 
> It looks like a comprehensive fix could be found in teaching DBD::Pg to
> use row caching:
> 
> https://rt.cpan.org/Public/Bug/Display.html?id=93266
> 
> 
> 
> On 03/10/2017 11:11 AM, Jason Stephenson wrote:
>> Hi, all.
>>
>> NOTE: This is not https://bugs.launchpad.net/evergreen/+bug/1671845. It
>> may be related or have a similar cause, but the experience/symptoms are
>> completely different.
>>
>> At this point, consider this a head's up, as well as a problem
>> description that I don't yet think I have enough information to file as
>> a bug report. It is also a request for anyone who wants to double check
>> this report and to help with debugging.
>>
>> I've noticed some bizarre behavior with DBI, MARC::Record, and writing
>> to a file with Perl version 5.20 and 5.22. (These versions of Perl ship
>> with Debian 8 Jessie and Ubuntu 16.04 Xenial Xerus, respectively.)
>>
>> I have a script (https://github.com/Dyrcona/boopsie) that I use to make
>> a weekly extract of records to send to Boopsie, Inc. on behalf of our
>> member libraries that use their app.
>>
>> What I have seen is that when run with the aforementioned versions of
>> Perl, the program consumes all of the RAM on the server and gets killed
>> by OOM killer. No output ever reaches the file. This suggests to me that
>> the problem occurs in the main loop with extract and converting the
>> MARCXML from the database, though it could be the Perl output buffer run
>> amok.
>>
>> The main loop of my program is similar, though less complicated, than
>> that of marc_export. I tried marc_export to see if it would have the
>> same problem. When extracting my whole database, it does:
>>
>> marc_export -a -e UTF-8 > all.mrc
>>
>> It also crashes if fed the output of an equivalent psql query to extract
>> all of the record ids, or if a file of all record ids is piped into
>> marc_export. It makes no difference if the output format is USMARC or
>> MARCXML.
>>
>> I can split this up into batches of 50,000 or so records (quite possibly
>> more) and all is well. I figured this out by dumping records for a
>> branch with around 51,000 items and that worked. My whole database has
>> just over 2.7 million, non-deleted bib records.
>>
>> This worked on Perl version 5.14 on Debian 7 Wheezy and on Ubuntu 14.04
>> Trusty Tahr.
>>
>> I hope to run marc_export with the Perl debugger to figure out the exact
>> cause. Until this is fixed, I'm using a work around in my scripts of
>> dumping MARCXML batches and converting them to USMARC and putting them
>> into 1 file with yaz-marcdump. This seems to work in light of the Lp bug
>> mentioned in the NOTE.
>>
>> Any and all information, contradictory or otherwise, from those using
>> Debian 8 or Ubuntu 16.04 is most welcome.
>>
>> Jason
>>