<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Thank you very much for all the
suggestions!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">As we do not use nginx (we use Apache),
we didn't try Jason's first suggestion and proceeded directly to
the second one - and it seems to have resolved the issue :-)!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">A more detailed report:</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">We are on Debian 10 without any
advanced security settings. It is a one-brick installation and
vandelay was originally directed to the original /tmp directory.
This didn't work. However, after changing to the newly created
directory /openils/temp which is fully owned by the opensrf user,
it has started working :-).</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thank you once again!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Linda<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2/4/21 7:12 PM, Daniel Wells wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGenA-Qy7DNuKGvkJYnUU2XzA=TV9aweR_fqpTkX6dwoyajRTw@mail.gmail.com">
<div dir="ltr">Jason's suggestions are very good. To add to that,
I'd be curious whether the unreadable file is there or not, and
what the permissions might be. Since the deletion of that file
happens after the point where it errors out for you, the file
should still be sitting there if you only have a read problem.
If it isn't there, then you probably have a write problem (or
a non-global "importer" dir), and your issue is actually
earlier.
<div><br>
</div>
<div>Naturally, you would want to test this relatively soon
after the error occurs.<br>
<div><br>
</div>
<div>Sincerely,</div>
<div>Dan</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Feb 4, 2021 at 12:54
PM Jason Stephenson <<a href="mailto:jason@sigio.com"
moz-do-not-send="true">jason@sigio.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote">Hello, all.<br>
<br>
Here are a few things to check:<br>
<br>
1. If you're using nginx as a proxy, and the upload routinely
fails on<br>
large files, check the client_max_body_size parameter in<br>
/etc/nginx/nginx.conf. Ours is set to 25m for 25 megabytes.<br>
<br>
2. In /openils/conf/opensrf.xml, lookf for<br>
open-ils.vandelay->app_settings->importer. Make sure
that<br>
<br>
a. the directory exists,<br>
b. it is hosted on shared storage, and<br>
c. it is accessible on all of your brick heads.<br>
<br>
It wouldn't hurt to make sure that it is not full. If it is
you can<br>
delete old files and may want to consider setting up a cron
job to<br>
routinely remove old files.<br>
<br>
3. Check the max_stanza_size in /etc/ejabberd/ejabberd.yml.
While it is<br>
supposed to no longer be necessary to change this setting
because of<br>
chunking and bundling, there are still some places where
chunking and<br>
bundling support is incomplete. If the above two options do
not work,<br>
you may want to try setting this to 2097152 or higher. You
will need to<br>
restart ejabberd, all OpenSRF servers, and Apache after making
this change.<br>
<br>
HtH,<br>
Jason<br>
<br>
On 2/4/21 11:51 AM, Tina Ji wrote:<br>
> <br>
> I encountered similar issues. It happens randomly with
files big or<br>
> small. I could not figure a pattern yet. It does not
appear to be a data<br>
> issue.<br>
> <br>
> <br>
> 1. Uploading stalled. Sometimes it may finish when trying
again with a<br>
> different MatchSet, not always. Sometimes I had to break
the file into<br>
> smaller chunks or recompile the file (MARCEdit).
Eventually the records<br>
> were uploaded and imported.<br>
> <br>
> 2. Uploading appears to be complete (100%), but some
records are not<br>
> loaded into the queue. I had to count the records in a
file and compare<br>
> it with the number of records in a queue every time.<br>
> <br>
> 3. Importing shows as complete, but some records are not
imported.<br>
> Trying to import those records again within the queue
never succeed. (I<br>
> encountered another issue here: export non-imported
records stopped<br>
> working for me. I am not sure it's a local issue or not
yet.<br>
> T00_MANY_REDIRECT). Uploading and importing those few
records in a<br>
> separate queue is successful.<br>
> <br>
> We were on 3.5, then recently rolled up to 3.5.1ish.<br>
> <br>
> <br>
> <br>
> Quoting Trevor Johnson <<a
href="mailto:tjohnson@apls.state.al.us" target="_blank"
moz-do-not-send="true">tjohnson@apls.state.al.us</a>>:<br>
> <br>
>> Not sure why but this issue happened with us when we
upgraded our<br>
>> server to Ubuntu 18.04(bionic). We’re still trying to
figure it out.<br>
>><br>
>> From: Evergreen-general<br>
>> <<a
href="mailto:evergreen-general-bounces@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true">evergreen-general-bounces@list.evergreen-ils.org</a>>
on behalf of Linda<br>
>> Jansová <<a href="mailto:linda.jansova@gmail.com"
target="_blank" moz-do-not-send="true">linda.jansova@gmail.com</a>><br>
>> Reply-To: Evergreen Discussion Group<br>
>> <<a
href="mailto:evergreen-general@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true">evergreen-general@list.evergreen-ils.org</a>><br>
>> Date: Thursday, February 4, 2021 at 2:00 AM<br>
>> To: "<a
href="mailto:evergreen-general@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true">evergreen-general@list.evergreen-ils.org</a>"<br>
>> <<a
href="mailto:evergreen-general@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true">evergreen-general@list.evergreen-ils.org</a>><br>
>> Subject: [Evergreen-general] MARC batch import not
processing uploaded<br>
>> files (in 3.5.1)<br>
>><br>
>><br>
>> Dear all,<br>
>><br>
>> We seem to have an issue with the MARC batch import
feature in<br>
>> Evergreen 3.5.1.<br>
>><br>
>> To make sure it is not caused by data in incorrect
format, we have –<br>
>> for testing purposes – used a MARCXML file exported
from the same<br>
>> Evergreen instance where we have encountered the
issue (the bib record<br>
>> has been exported using the web client MARC batch
export feature).<br>
>> This XML file is attached.<br>
>><br>
>> We have tried to import the file using admin accounts
to the following<br>
>> community/test servers:<br>
>><br>
>> * <a
href="https://demo.evergreencatalog.com/eg/staff/home"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://demo.evergreencatalog.com/eg/staff/home</a>
(3.6.0) – works<br>
>> okay<br>
>> * <a
href="https://bugsquash.mobiusconsortium.org/eg/staff"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://bugsquash.mobiusconsortium.org/eg/staff</a>
(3.5.0) – works<br>
>> okay<br>
>> * our community server<br>
>> <a
href="https://evergreendemo.jabok.cuni.cz/eg/staff/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://evergreendemo.jabok.cuni.cz/eg/staff/</a>
(3.5.1) – does not work<br>
>> okay<br>
>> * our another test (3.5.1) – does not work okay<br>
>><br>
>> While when it works okay, the file is processed and
all three stages<br>
>> (upload, enqueue and import) progress bars go to a
100 % point, in our<br>
>> 3.5.1 instances we have ended up with the first
progress bar. I'm also<br>
>> attaching a screenshot from the Mobius community test
server and from<br>
>> ours to clarify what I mean.<br>
>><br>
>> Using the opensrf log we have been able to find out
the cause of the<br>
>> issue as we were getting error messages like this
one:<br>
>><br>
>> [2021-02-01 21:34:29] open-ils.vandelay [ERR<br>
>> :7692:Vandelay.pm:272:1612211178912619] unable to
read MARC file<br>
>> /tmp/56feb3dd5296fc1f8e579bdc8b29dca7.mrc<br>
>><br>
>> These point to the 272nd line of Vandelay.pm<br>
>> (<a
href="https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Vandelay.pm;hb=43a802ae2c56c9342bfd3a6a4556292d00760d3e</a>).<br>
>><br>
>><br>
>> We are unsure whether it may be a permissions issue
(if so, does<br>
>> anyone happen to know a complete list of Evergreen
permissions needed<br>
>> for the MARC batch import to run and go through all
the stages of data<br>
>> processing?) or something else entirely.<br>
>><br>
>> Thank you in advance for any advice!<br>
>><br>
>> Linda<br>
> <br>
> <br>
_______________________________________________<br>
Evergreen-general mailing list<br>
<a href="mailto:Evergreen-general@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true">Evergreen-general@list.evergreen-ils.org</a><br>
<a
href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Evergreen-general mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Evergreen-general@list.evergreen-ils.org">Evergreen-general@list.evergreen-ils.org</a>
<a class="moz-txt-link-freetext" href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general</a>
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>