[Evergreen-general] MARC batch import not processing uploaded files (in 3.5.1)

Thu Feb 4 15:46:25 EST 2021

Thanks so much, Jason. I should have copied the whole lot between Jeff and me.

It's really random and also happens to small files. I checked the  
console a few times. Nothing rang a bell. Sorry, I did not capture it.

That being said, I'd love to test the new server when available.

-----------------------------------------
Thanks, Jeff. That confirmed the impression I had. It does not look to  
me it is related to file size. I encountered the issue with small  
files like having 20 records. Tina

Quoting Jeff Davis <jeff.davis at bc.libraries.coop>:

[Hide Quoted Text]
Thanks for forwarding!  I took a look and I think our environment is OK:

1. We set client_max_body_size to 10M in nginx config.

2. The upload folder is /srv/openils/var/tmp which is shared and
accessible on all app servers.  It's starting to get full but still has
lots of room available.

3. We set max_stanza_size to 2000000 in ejabberd config.

A couple of our values are slightly smaller than Jason's, but I don't
think that's a problem -- our settings are still pretty generous, and
our MARC import issues are not limited to large files.  We can try
increasing the upload size in nginx if you think it would help, though.

I also took a look for "unable to read MARC file" errors in our logs.  I
found only a couple of instances in the past three months, always when
uploading via Acq, and there is no filename in the log message (possibly
someone clicked "Upload" without selecting a file).  So I don't think
our problem is the same as Linda's.
-------------------------

Tina

Quoting Jason Stephenson <jason at sigio.com>:

> Tina,
>
> You could have Jeff try increasing some of those settings to see if that
> helps.
>
> Apache 2.4 also has a few settings for limiting request sizes. These are
> not usually changed from the defaults on Debian/Ubuntu.  The default is 2GB.
>
> You might want to have Jeff check if LimitRequestBody is set anywhere in
> the Apache configuration files.
>
> Barring that being an issue, I'd check the JavaScript console log in the
> brower's developer tools for errors while doing the imports.  I'd check
> the OpenSRF logs for errors or messages related to the open-ils.vandelay
> service. It could be that you've encountered a bug or have some bad code.
>
> We have a test server set up with Evergreen 3.5.2 in preparation for an
> upgrade later this year. I'll check with our cataloger to see if she has
> encountered any issues with bib imports on the test server.
>
> HtH,
> Jason
>
> On 2/4/21 2:03 PM, Tina Ji wrote:
>> Thanks, Jason.
>>
>> Here is the info of our server given by Jeff:
>>
>>
>> 1. We set client_max_body_size to 10M in nginx config.
>>
>> 2. The upload folder is /srv/openils/var/tmp which is shared and
>> accessible on all app servers.  It's starting to get full but still has
>> lots of room available.
>>
>> 3. We set max_stanza_size to 2000000 in ejabberd config.
>>
>>
>>
>>
>> Quoting Jason Stephenson <jason at sigio.com>:
>>
>>> Hello, all.
>>>
>>> Here are a few things to check:
>>>
>>> 1. If you're using nginx as a proxy, and the upload routinely fails on
>>> large files, check the client_max_body_size parameter in
>>> /etc/nginx/nginx.conf. Ours is set to 25m for 25 megabytes.
>>>
>>> 2. In /openils/conf/opensrf.xml, lookf for
>>> open-ils.vandelay->app_settings->importer. Make sure that
>>>
>>> a. the directory exists,
>>> b. it is hosted on shared storage, and
>>> c. it is accessible on all of your brick heads.
>>>
>>> It wouldn't hurt to make sure that it is not full. If it is you can
>>> delete old files and may want to consider setting up a cron job to
>>> routinely remove old files.
>>>
>>> 3. Check the max_stanza_size in /etc/ejabberd/ejabberd.yml. While it is
>>> supposed to no longer be necessary to change this setting because of
>>> chunking and bundling, there are still some places where chunking and
>>> bundling support is incomplete. If the above two options do not work,
>>> you may want to try setting this to 2097152 or higher. You will need to
>>> restart ejabberd, all OpenSRF servers, and Apache after making this
>>> change.
>>>
>>> HtH,
>>> Jason
>>>
>>> On 2/4/21 11:51 AM, Tina Ji wrote:
>>>>
>>>> I encountered similar issues. It happens randomly with files big or
>>>> small. I could not figure a pattern yet. It does not appear to be a data
>>>> issue.
>>>>
>>>>
>>>> 1. Uploading stalled. Sometimes it may finish when trying again with a
>>>> different MatchSet, not always. Sometimes I had to break the file into
>>>> smaller chunks or recompile the file (MARCEdit). Eventually the records
>>>> were uploaded and imported.
>>>>
>>>> 2. Uploading appears to be complete (100%), but some records are not
>>>> loaded into the queue. I had to count the records in a file and compare
>>>> it with the number of records in a queue every time.
>>>>
>>>> 3. Importing shows as complete, but some records are not imported.
>>>> Trying to import those records again within the queue never succeed. (I
>>>> encountered another issue here: export non-imported records stopped
>>>> working for me. I am not sure it's a local issue or not yet.
>>>> T00_MANY_REDIRECT). Uploading and importing those few records in a
>>>> separate queue is successful.
>>>>
>>>> We were on 3.5, then recently rolled up to 3.5.1ish.
>>>>
>>>>
>>>>
>>>> Quoting Trevor Johnson <tjohnson at apls.state.al.us>:
>>>>
>>>>> Not sure why but this issue happened with us when we upgraded our
>>>>> server to Ubuntu 18.04(bionic). We’re still trying to figure it out.
>>>>>
>>>>> From: Evergreen-general
>>>>> <evergreen-general-bounces at list.evergreen-ils.org> on behalf of Linda
>>>>> Jansová <linda.jansova at gmail.com>
>>>>> Reply-To: Evergreen Discussion Group
>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>> Date: Thursday, February 4, 2021 at 2:00 AM
>>>>> To: "evergreen-general at list.evergreen-ils.org"
>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>> Subject: [Evergreen-general] MARC batch import not processing uploaded
>>>>> files (in 3.5.1)
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> We seem to have an issue with the MARC batch import feature in
>>>>> Evergreen 3.5.1.
>>>>>
>>>>> To make sure it is not caused by data in incorrect format, we have –
>>>>> for testing purposes – used a MARCXML file exported from the same
>>>>> Evergreen instance where we have encountered the issue (the bib record
>>>>> has been exported using the web client MARC batch export feature).
>>>>> This XML file is attached.
>>>>>
>>>>> We have tried to import the file using admin accounts to the following
>>>>> community/test servers:
>>>>>
>>>>>   *   https://demo.evergreencatalog.com/eg/staff/home (3.6.0) – works
>>>>> okay
>>>>>   *   https://bugsquash.mobiusconsortium.org/eg/staff (3.5.0) – works
>>>>> okay
>>>>>   *   our community server
>>>>> https://evergreendemo.jabok.cuni.cz/eg/staff/ (3.5.1) – does not work
>>>>> okay
>>>>>   *   our another test (3.5.1) – does not work okay
>>>>>
>>>>> While when it works okay, the file is processed and all three stages
>>>>> (upload, enqueue and import) progress bars go to a 100 % point, in our
>>>>> 3.5.1 instances we have ended up with the first progress bar. I'm also
>>>>> attaching a screenshot from the Mobius community test server and from
>>>>> ours to clarify what I mean.
>>>>>
>>>>> Using the opensrf log we have been able to find out the cause of the
>>>>> issue as we were getting error messages like this one:
>>>>>
>>>>> [2021-02-01 21:34:29] open-ils.vandelay [ERR
>>>>> :7692:Vandelay.pm:272:1612211178912619] unable to read MARC file
>>>>> /tmp/56feb3dd5296fc1f8e579bdc8b29dca7.mrc
>>>>>
>>>>> These point to the 272nd line of Vandelay.pm
>>>>> (https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e).
>>>>>
>>>>>
>>>>>
>>>>> We are unsure whether it may be a permissions issue (if so, does
>>>>> anyone happen to know a complete list of Evergreen permissions needed
>>>>> for the MARC batch import to run and go through all the stages of data
>>>>> processing?) or something else entirely.
>>>>>
>>>>> Thank you in advance for any advice!
>>>>>
>>>>> Linda
>>>>
>>>>
>>> _______________________________________________
>>> Evergreen-general mailing list
>>> Evergreen-general at list.evergreen-ils.org
>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
>>
>>
> _______________________________________________
> Evergreen-general mailing list
> Evergreen-general at list.evergreen-ils.org
> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general

-- 
Tina Ji
1-888-848-9250 ext 1014
Support Specialist
BC Libraries Co-operative