[Evergreen-general] MARC batch import not processing uploaded files (in 3.5.1)

Jason Stephenson jason at sigio.com
Thu Feb 4 17:21:41 EST 2021 

Tina,

Our cataloger did some uploads on our test server with 3.5.2 and
reported no issues.

At this point, I don't know what your problem is, but looking for
open-ils.vandelay messages in the log files is where I'd go next.

Jason

On 2/4/21 3:46 PM, Tina Ji wrote:
> 
> Thanks so much, Jason. I should have copied the whole lot between Jeff
> and me.
> 
> It's really random and also happens to small files. I checked the
> console a few times. Nothing rang a bell. Sorry, I did not capture it.
> 
> That being said, I'd love to test the new server when available.
> 
> -----------------------------------------
> Thanks, Jeff. That confirmed the impression I had. It does not look to
> me it is related to file size. I encountered the issue with small files
> like having 20 records. Tina
> 
> 
> Quoting Jeff Davis <jeff.davis at bc.libraries.coop>:
> 
> [Hide Quoted Text]
> Thanks for forwarding!  I took a look and I think our environment is OK:
> 
> 1. We set client_max_body_size to 10M in nginx config.
> 
> 2. The upload folder is /srv/openils/var/tmp which is shared and
> accessible on all app servers.  It's starting to get full but still has
> lots of room available.
> 
> 3. We set max_stanza_size to 2000000 in ejabberd config.
> 
> A couple of our values are slightly smaller than Jason's, but I don't
> think that's a problem -- our settings are still pretty generous, and
> our MARC import issues are not limited to large files.  We can try
> increasing the upload size in nginx if you think it would help, though.
> 
> I also took a look for "unable to read MARC file" errors in our logs.  I
> found only a couple of instances in the past three months, always when
> uploading via Acq, and there is no filename in the log message (possibly
> someone clicked "Upload" without selecting a file).  So I don't think
> our problem is the same as Linda's.
> -------------------------
> 
> Tina
> 
> Quoting Jason Stephenson <jason at sigio.com>:
> 
>> Tina,
>>
>> You could have Jeff try increasing some of those settings to see if that
>> helps.
>>
>> Apache 2.4 also has a few settings for limiting request sizes. These are
>> not usually changed from the defaults on Debian/Ubuntu.  The default
>> is 2GB.
>>
>> You might want to have Jeff check if LimitRequestBody is set anywhere in
>> the Apache configuration files.
>>
>> Barring that being an issue, I'd check the JavaScript console log in the
>> brower's developer tools for errors while doing the imports.  I'd check
>> the OpenSRF logs for errors or messages related to the open-ils.vandelay
>> service. It could be that you've encountered a bug or have some bad code.
>>
>> We have a test server set up with Evergreen 3.5.2 in preparation for an
>> upgrade later this year. I'll check with our cataloger to see if she has
>> encountered any issues with bib imports on the test server.
>>
>> HtH,
>> Jason
>>
>> On 2/4/21 2:03 PM, Tina Ji wrote:
>>> Thanks, Jason.
>>>
>>> Here is the info of our server given by Jeff:
>>>
>>>
>>> 1. We set client_max_body_size to 10M in nginx config.
>>>
>>> 2. The upload folder is /srv/openils/var/tmp which is shared and
>>> accessible on all app servers.  It's starting to get full but still has
>>> lots of room available.
>>>
>>> 3. We set max_stanza_size to 2000000 in ejabberd config.
>>>
>>>
>>>
>>>
>>> Quoting Jason Stephenson <jason at sigio.com>:
>>>
>>>> Hello, all.
>>>>
>>>> Here are a few things to check:
>>>>
>>>> 1. If you're using nginx as a proxy, and the upload routinely fails on
>>>> large files, check the client_max_body_size parameter in
>>>> /etc/nginx/nginx.conf. Ours is set to 25m for 25 megabytes.
>>>>
>>>> 2. In /openils/conf/opensrf.xml, lookf for
>>>> open-ils.vandelay->app_settings->importer. Make sure that
>>>>
>>>> a. the directory exists,
>>>> b. it is hosted on shared storage, and
>>>> c. it is accessible on all of your brick heads.
>>>>
>>>> It wouldn't hurt to make sure that it is not full. If it is you can
>>>> delete old files and may want to consider setting up a cron job to
>>>> routinely remove old files.
>>>>
>>>> 3. Check the max_stanza_size in /etc/ejabberd/ejabberd.yml. While it is
>>>> supposed to no longer be necessary to change this setting because of
>>>> chunking and bundling, there are still some places where chunking and
>>>> bundling support is incomplete. If the above two options do not work,
>>>> you may want to try setting this to 2097152 or higher. You will need to
>>>> restart ejabberd, all OpenSRF servers, and Apache after making this
>>>> change.
>>>>
>>>> HtH,
>>>> Jason
>>>>
>>>> On 2/4/21 11:51 AM, Tina Ji wrote:
>>>>>
>>>>> I encountered similar issues. It happens randomly with files big or
>>>>> small. I could not figure a pattern yet. It does not appear to be a
>>>>> data
>>>>> issue.
>>>>>
>>>>>
>>>>> 1. Uploading stalled. Sometimes it may finish when trying again with a
>>>>> different MatchSet, not always. Sometimes I had to break the file into
>>>>> smaller chunks or recompile the file (MARCEdit). Eventually the
>>>>> records
>>>>> were uploaded and imported.
>>>>>
>>>>> 2. Uploading appears to be complete (100%), but some records are not
>>>>> loaded into the queue. I had to count the records in a file and
>>>>> compare
>>>>> it with the number of records in a queue every time.
>>>>>
>>>>> 3. Importing shows as complete, but some records are not imported.
>>>>> Trying to import those records again within the queue never
>>>>> succeed. (I
>>>>> encountered another issue here: export non-imported records stopped
>>>>> working for me. I am not sure it's a local issue or not yet.
>>>>> T00_MANY_REDIRECT). Uploading and importing those few records in a
>>>>> separate queue is successful.
>>>>>
>>>>> We were on 3.5, then recently rolled up to 3.5.1ish.
>>>>>
>>>>>
>>>>>
>>>>> Quoting Trevor Johnson <tjohnson at apls.state.al.us>:
>>>>>
>>>>>> Not sure why but this issue happened with us when we upgraded our
>>>>>> server to Ubuntu 18.04(bionic). We’re still trying to figure it out.
>>>>>>
>>>>>> From: Evergreen-general
>>>>>> <evergreen-general-bounces at list.evergreen-ils.org> on behalf of Linda
>>>>>> Jansová <linda.jansova at gmail.com>
>>>>>> Reply-To: Evergreen Discussion Group
>>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>>> Date: Thursday, February 4, 2021 at 2:00 AM
>>>>>> To: "evergreen-general at list.evergreen-ils.org"
>>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>>> Subject: [Evergreen-general] MARC batch import not processing
>>>>>> uploaded
>>>>>> files (in 3.5.1)
>>>>>>
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> We seem to have an issue with the MARC batch import feature in
>>>>>> Evergreen 3.5.1.
>>>>>>
>>>>>> To make sure it is not caused by data in incorrect format, we have –
>>>>>> for testing purposes – used a MARCXML file exported from the same
>>>>>> Evergreen instance where we have encountered the issue (the bib
>>>>>> record
>>>>>> has been exported using the web client MARC batch export feature).
>>>>>> This XML file is attached.
>>>>>>
>>>>>> We have tried to import the file using admin accounts to the
>>>>>> following
>>>>>> community/test servers:
>>>>>>
>>>>>>   *   https://demo.evergreencatalog.com/eg/staff/home (3.6.0) – works
>>>>>> okay
>>>>>>   *   https://bugsquash.mobiusconsortium.org/eg/staff (3.5.0) – works
>>>>>> okay
>>>>>>   *   our community server
>>>>>> https://evergreendemo.jabok.cuni.cz/eg/staff/ (3.5.1) – does not work
>>>>>> okay
>>>>>>   *   our another test (3.5.1) – does not work okay
>>>>>>
>>>>>> While when it works okay, the file is processed and all three stages
>>>>>> (upload, enqueue and import) progress bars go to a 100 % point, in
>>>>>> our
>>>>>> 3.5.1 instances we have ended up with the first progress bar. I'm
>>>>>> also
>>>>>> attaching a screenshot from the Mobius community test server and from
>>>>>> ours to clarify what I mean.
>>>>>>
>>>>>> Using the opensrf log we have been able to find out the cause of the
>>>>>> issue as we were getting error messages like this one:
>>>>>>
>>>>>> [2021-02-01 21:34:29] open-ils.vandelay [ERR
>>>>>> :7692:Vandelay.pm:272:1612211178912619] unable to read MARC file
>>>>>> /tmp/56feb3dd5296fc1f8e579bdc8b29dca7.mrc
>>>>>>
>>>>>> These point to the 272nd line of Vandelay.pm
>>>>>> (https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> We are unsure whether it may be a permissions issue (if so, does
>>>>>> anyone happen to know a complete list of Evergreen permissions needed
>>>>>> for the MARC batch import to run and go through all the stages of
>>>>>> data
>>>>>> processing?) or something else entirely.
>>>>>>
>>>>>> Thank you in advance for any advice!
>>>>>>
>>>>>> Linda
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Evergreen-general mailing list
>>>> Evergreen-general at list.evergreen-ils.org
>>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
>>>>
>>>
>>>
>> _______________________________________________
>> Evergreen-general mailing list
>> Evergreen-general at list.evergreen-ils.org
>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
> 
>

More information about the Evergreen-general mailing list