[Evergreen-general] MARC batch import not processing uploaded files (in 3.5.1)

Tina Ji tina.ji at bc.libraries.coop
Fri Feb 5 12:21:43 EST 2021 

Thanks, Jason. I will ask Jeff to look into the log file when I  
encounter the issue again. Tina


Quoting Jason Stephenson <jason at sigio.com>:

> Tina,
>
> Our cataloger did some uploads on our test server with 3.5.2 and
> reported no issues.
>
> At this point, I don't know what your problem is, but looking for
> open-ils.vandelay messages in the log files is where I'd go next.
>
> Jason
>
> On 2/4/21 3:46 PM, Tina Ji wrote:
>>
>> Thanks so much, Jason. I should have copied the whole lot between Jeff
>> and me.
>>
>> It's really random and also happens to small files. I checked the
>> console a few times. Nothing rang a bell. Sorry, I did not capture it.
>>
>> That being said, I'd love to test the new server when available.
>>
>> -----------------------------------------
>> Thanks, Jeff. That confirmed the impression I had. It does not look to
>> me it is related to file size. I encountered the issue with small files
>> like having 20 records. Tina
>>
>>
>> Quoting Jeff Davis <jeff.davis at bc.libraries.coop>:
>>
>> [Hide Quoted Text]
>> Thanks for forwarding!  I took a look and I think our environment is OK:
>>
>> 1. We set client_max_body_size to 10M in nginx config.
>>
>> 2. The upload folder is /srv/openils/var/tmp which is shared and
>> accessible on all app servers.  It's starting to get full but still has
>> lots of room available.
>>
>> 3. We set max_stanza_size to 2000000 in ejabberd config.
>>
>> A couple of our values are slightly smaller than Jason's, but I don't
>> think that's a problem -- our settings are still pretty generous, and
>> our MARC import issues are not limited to large files.  We can try
>> increasing the upload size in nginx if you think it would help, though.
>>
>> I also took a look for "unable to read MARC file" errors in our logs.  I
>> found only a couple of instances in the past three months, always when
>> uploading via Acq, and there is no filename in the log message (possibly
>> someone clicked "Upload" without selecting a file).  So I don't think
>> our problem is the same as Linda's.
>> -------------------------
>>
>> Tina
>>
>> Quoting Jason Stephenson <jason at sigio.com>:
>>
>>> Tina,
>>>
>>> You could have Jeff try increasing some of those settings to see if that
>>> helps.
>>>
>>> Apache 2.4 also has a few settings for limiting request sizes. These are
>>> not usually changed from the defaults on Debian/Ubuntu.  The default
>>> is 2GB.
>>>
>>> You might want to have Jeff check if LimitRequestBody is set anywhere in
>>> the Apache configuration files.
>>>
>>> Barring that being an issue, I'd check the JavaScript console log in the
>>> brower's developer tools for errors while doing the imports.  I'd check
>>> the OpenSRF logs for errors or messages related to the open-ils.vandelay
>>> service. It could be that you've encountered a bug or have some bad code.
>>>
>>> We have a test server set up with Evergreen 3.5.2 in preparation for an
>>> upgrade later this year. I'll check with our cataloger to see if she has
>>> encountered any issues with bib imports on the test server.
>>>
>>> HtH,
>>> Jason
>>>
>>> On 2/4/21 2:03 PM, Tina Ji wrote:
>>>> Thanks, Jason.
>>>>
>>>> Here is the info of our server given by Jeff:
>>>>
>>>>
>>>> 1. We set client_max_body_size to 10M in nginx config.
>>>>
>>>> 2. The upload folder is /srv/openils/var/tmp which is shared and
>>>> accessible on all app servers.  It's starting to get full but still has
>>>> lots of room available.
>>>>
>>>> 3. We set max_stanza_size to 2000000 in ejabberd config.
>>>>
>>>>
>>>>
>>>>
>>>> Quoting Jason Stephenson <jason at sigio.com>:
>>>>
>>>>> Hello, all.
>>>>>
>>>>> Here are a few things to check:
>>>>>
>>>>> 1. If you're using nginx as a proxy, and the upload routinely fails on
>>>>> large files, check the client_max_body_size parameter in
>>>>> /etc/nginx/nginx.conf. Ours is set to 25m for 25 megabytes.
>>>>>
>>>>> 2. In /openils/conf/opensrf.xml, lookf for
>>>>> open-ils.vandelay->app_settings->importer. Make sure that
>>>>>
>>>>> a. the directory exists,
>>>>> b. it is hosted on shared storage, and
>>>>> c. it is accessible on all of your brick heads.
>>>>>
>>>>> It wouldn't hurt to make sure that it is not full. If it is you can
>>>>> delete old files and may want to consider setting up a cron job to
>>>>> routinely remove old files.
>>>>>
>>>>> 3. Check the max_stanza_size in /etc/ejabberd/ejabberd.yml. While it is
>>>>> supposed to no longer be necessary to change this setting because of
>>>>> chunking and bundling, there are still some places where chunking and
>>>>> bundling support is incomplete. If the above two options do not work,
>>>>> you may want to try setting this to 2097152 or higher. You will need to
>>>>> restart ejabberd, all OpenSRF servers, and Apache after making this
>>>>> change.
>>>>>
>>>>> HtH,
>>>>> Jason
>>>>>
>>>>> On 2/4/21 11:51 AM, Tina Ji wrote:
>>>>>>
>>>>>> I encountered similar issues. It happens randomly with files big or
>>>>>> small. I could not figure a pattern yet. It does not appear to be a
>>>>>> data
>>>>>> issue.
>>>>>>
>>>>>>
>>>>>> 1. Uploading stalled. Sometimes it may finish when trying again with a
>>>>>> different MatchSet, not always. Sometimes I had to break the file into
>>>>>> smaller chunks or recompile the file (MARCEdit). Eventually the
>>>>>> records
>>>>>> were uploaded and imported.
>>>>>>
>>>>>> 2. Uploading appears to be complete (100%), but some records are not
>>>>>> loaded into the queue. I had to count the records in a file and
>>>>>> compare
>>>>>> it with the number of records in a queue every time.
>>>>>>
>>>>>> 3. Importing shows as complete, but some records are not imported.
>>>>>> Trying to import those records again within the queue never
>>>>>> succeed. (I
>>>>>> encountered another issue here: export non-imported records stopped
>>>>>> working for me. I am not sure it's a local issue or not yet.
>>>>>> T00_MANY_REDIRECT). Uploading and importing those few records in a
>>>>>> separate queue is successful.
>>>>>>
>>>>>> We were on 3.5, then recently rolled up to 3.5.1ish.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Quoting Trevor Johnson <tjohnson at apls.state.al.us>:
>>>>>>
>>>>>>> Not sure why but this issue happened with us when we upgraded our
>>>>>>> server to Ubuntu 18.04(bionic). We’re still trying to figure it out.
>>>>>>>
>>>>>>> From: Evergreen-general
>>>>>>> <evergreen-general-bounces at list.evergreen-ils.org> on behalf of Linda
>>>>>>> Jansová <linda.jansova at gmail.com>
>>>>>>> Reply-To: Evergreen Discussion Group
>>>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>>>> Date: Thursday, February 4, 2021 at 2:00 AM
>>>>>>> To: "evergreen-general at list.evergreen-ils.org"
>>>>>>> <evergreen-general at list.evergreen-ils.org>
>>>>>>> Subject: [Evergreen-general] MARC batch import not processing
>>>>>>> uploaded
>>>>>>> files (in 3.5.1)
>>>>>>>
>>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> We seem to have an issue with the MARC batch import feature in
>>>>>>> Evergreen 3.5.1.
>>>>>>>
>>>>>>> To make sure it is not caused by data in incorrect format, we have –
>>>>>>> for testing purposes – used a MARCXML file exported from the same
>>>>>>> Evergreen instance where we have encountered the issue (the bib
>>>>>>> record
>>>>>>> has been exported using the web client MARC batch export feature).
>>>>>>> This XML file is attached.
>>>>>>>
>>>>>>> We have tried to import the file using admin accounts to the
>>>>>>> following
>>>>>>> community/test servers:
>>>>>>>
>>>>>>>   *   https://demo.evergreencatalog.com/eg/staff/home (3.6.0) – works
>>>>>>> okay
>>>>>>>   *   https://bugsquash.mobiusconsortium.org/eg/staff (3.5.0) – works
>>>>>>> okay
>>>>>>>   *   our community server
>>>>>>> https://evergreendemo.jabok.cuni.cz/eg/staff/ (3.5.1) – does not work
>>>>>>> okay
>>>>>>>   *   our another test (3.5.1) – does not work okay
>>>>>>>
>>>>>>> While when it works okay, the file is processed and all three stages
>>>>>>> (upload, enqueue and import) progress bars go to a 100 % point, in
>>>>>>> our
>>>>>>> 3.5.1 instances we have ended up with the first progress bar. I'm
>>>>>>> also
>>>>>>> attaching a screenshot from the Mobius community test server and from
>>>>>>> ours to clarify what I mean.
>>>>>>>
>>>>>>> Using the opensrf log we have been able to find out the cause of the
>>>>>>> issue as we were getting error messages like this one:
>>>>>>>
>>>>>>> [2021-02-01 21:34:29] open-ils.vandelay [ERR
>>>>>>> :7692:Vandelay.pm:272:1612211178912619] unable to read MARC file
>>>>>>> /tmp/56feb3dd5296fc1f8e579bdc8b29dca7.mrc
>>>>>>>
>>>>>>> These point to the 272nd line of Vandelay.pm
>>>>>>> (https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We are unsure whether it may be a permissions issue (if so, does
>>>>>>> anyone happen to know a complete list of Evergreen permissions needed
>>>>>>> for the MARC batch import to run and go through all the stages of
>>>>>>> data
>>>>>>> processing?) or something else entirely.
>>>>>>>
>>>>>>> Thank you in advance for any advice!
>>>>>>>
>>>>>>> Linda
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Evergreen-general mailing list
>>>>> Evergreen-general at list.evergreen-ils.org
>>>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Evergreen-general mailing list
>>> Evergreen-general at list.evergreen-ils.org
>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
>>
>>
> _______________________________________________
> Evergreen-general mailing list
> Evergreen-general at list.evergreen-ils.org
> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general


-- 
Tina Ji
1-888-848-9250 ext 1014
Support Specialist
BC Libraries Co-operative

More information about the Evergreen-general mailing list