[Evergreen-general] MARC batch import not processing uploaded files (in 3.5.1)

[Tina Ji]

I encountered similar issues. It happens randomly with files big or  
small. I could not figure a pattern yet. It does not appear to be a  
data issue.


1. Uploading stalled. Sometimes it may finish when trying again with a  
different MatchSet, not always. Sometimes I had to break the file into  
smaller chunks or recompile the file (MARCEdit). Eventually the  
records were uploaded and imported.

2. Uploading appears to be complete (100%), but some records are not  
loaded into the queue. I had to count the records in a file and  
compare it with the number of records in a queue every time.

3. Importing shows as complete, but some records are not imported.  
Trying to import those records again within the queue never succeed.  
(I encountered another issue here: export non-imported records stopped  
working for me. I am not sure it's a local issue or not yet.  
T00_MANY_REDIRECT). Uploading and importing those few records in a  
separate queue is successful.

We were on 3.5, then recently rolled up to 3.5.1ish.



Quoting Trevor Johnson <tjohnson at apls.state.al.us>:

> Not sure why but this issue happened with us when we upgraded our  
> server to Ubuntu 18.04(bionic). We’re still trying to figure it out.
>
> From: Evergreen-general  
> <evergreen-general-bounces at list.evergreen-ils.org> on behalf of  
> Linda Jansová <linda.jansova at gmail.com>
> Reply-To: Evergreen Discussion Group  
> <evergreen-general at list.evergreen-ils.org>
> Date: Thursday, February 4, 2021 at 2:00 AM
> To: "evergreen-general at list.evergreen-ils.org"  
> <evergreen-general at list.evergreen-ils.org>
> Subject: [Evergreen-general] MARC batch import not processing  
> uploaded files (in 3.5.1)
>
>
> Dear all,
>
> We seem to have an issue with the MARC batch import feature in  
> Evergreen 3.5.1.
>
> To make sure it is not caused by data in incorrect format, we have –  
> for testing purposes – used a MARCXML file exported from the same  
> Evergreen instance where we have encountered the issue (the bib  
> record has been exported using the web client MARC batch export  
> feature). This XML file is attached.
>
> We have tried to import the file using admin accounts to the  
> following community/test servers:
>
>   *   https://demo.evergreencatalog.com/eg/staff/home (3.6.0) – works okay
>   *   https://bugsquash.mobiusconsortium.org/eg/staff (3.5.0) – works okay
>   *   our community server  
> https://evergreendemo.jabok.cuni.cz/eg/staff/ (3.5.1) – does not  
> work okay
>   *   our another test (3.5.1) – does not work okay
>
> While when it works okay, the file is processed and all three stages  
> (upload, enqueue and import) progress bars go to a 100 % point, in  
> our 3.5.1 instances we have ended up with the first progress bar.  
> I'm also attaching a screenshot from the Mobius community test  
> server and from ours to clarify what I mean.
>
> Using the opensrf log we have been able to find out the cause of the  
> issue as we were getting error messages like this one:
>
> [2021-02-01 21:34:29] open-ils.vandelay [ERR  
> :7692:Vandelay.pm:272:1612211178912619] unable to read MARC file  
> /tmp/56feb3dd5296fc1f8e579bdc8b29dca7.mrc
>
> These point to the 272nd line of Vandelay.pm  
> (https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e).
>
> We are unsure whether it may be a permissions issue (if so, does  
> anyone happen to know a complete list of Evergreen permissions  
> needed for the MARC batch import to run and go through all the  
> stages of data processing?) or something else entirely.
>
> Thank you in advance for any advice!
>
> Linda


-- 
Tina Ji
1-888-848-9250 ext 1014
Support Specialist
BC Libraries Co-operative