[OPEN-ILS-GENERAL] Upgrading from 2.10 to 2.11 error questions

[Jason Stephenson]

Josh,

Unfortunately, there are parts of the ingest process that cannot be done
in parallel, and pingest only does a subset of a full ingest anyway.

What I am now working on will update the biblio.record_entry in such a
way that a full ingest will be done, including call the maintain_901
function in the database (rather than duplicating it in the script).
This involves messing with internal flags and a plethora of command line
options to skip various portions of the ingest. This version, which I am
calling ngingest at the moment, cannot be parallelized without a great
deal more smarts.

I will share it to my Evergreen utilities repository on github when I'm
convinced it is working.

Cheers,
Jason

On 04/20/2017 12:34 PM, Josh Stompro wrote:
> Jason, I would be interested in seeing your script.  I wonder if your pingest.pl script that many people use for reingest could be extended with an option to trigger the maintain_901 function as another method?  Some sort of "trigger bib update" option?  That could handle spreading out the load/locks also.  Maybe that wouldn't allow the user to see the encoding errors though?
> 
> Maybe the encoding errors should be tested for a different way though, instead of trying to combine that with this update?
> 
> Josh Stompro - LARL IT Director
> 
> 
> -----Original Message-----
> From: Open-ils-general [mailto:open-ils-general-bounces at list.georgialibraries.org] On Behalf Of Jason Stephenson
> Sent: Thursday, April 20, 2017 7:20 AM
> To: Evergreen Discussion Group
> Subject: Re: [OPEN-ILS-GENERAL] Upgrading from 2.10 to 2.11 error questions
> 
> Hi, Jesse.
> 
> I won't repeat what Dan said, but I'll add that with a change to an internal flag or two this update will also take care of any ingests required. I think there's a browse or facet ingest required for 2.11.
> 
> I'm preparing to upgrade from 2.10 to 2.12, and I've run into a similar issue as you with the maintain_901 function and some records with "bad"
> characters. I'm working on a Perl script to pull the MARC out of the database and do the maintain_901 work in a function in the script. The advantage of this is two fold:
> 
> #1 I can trap any errors that occur and output a warning message with the record id that causes the error so it can later be fixed.
> 
> #2 By modifying the MARC and then updating the record in the database, I can get the ingests to trigger without altering any database flags.
> 
> It won't be fast, but it won't have the downside of potentially locking significant portions of the biblio.record_entry table. Updating 1 row at a time should keep it at row locks, or page locks at worst.
> 
> If there is enough interest expressed on the list, I could share this code on github or elsewhere.
> 
> Cheers,
> Jason
> 
> On 04/19/2017 04:22 PM, Jesse McCarty wrote:
>> Hello Everyone,
>>
>>  
>>
>> We are preparing for our spring upgrade to Evergreen, moving from 
>> 2.10.3 to 2.11.3 and ran into one little bump. As part of the DB 
>> upgrade, there is an update setting the 901$s for bib records. First 
>> question, as seen in the attached screen shot, this threw a bunch of 
>> 'no mapping found for....' Errors. Can this be safely ignored and 
>> proceed with running the system after upgrading with no issues (we 
>> haven't seen any issue in our testing)?
>>
>>  
>>
>> The second, this update seem to take longer than 24 hours.  With that 
>> in mind would we be able to process the entire upgrade, then use 
>> Evergreen in daily production while this DB update finishes in the 
>> background? Or does this need to be 100% complete before allowing 
>> library's connection to the system?
>>
>>  
>>
>> Thanks in advance,
>>
>>  
>>
>> Jesse McCarty
>>
>> City of Burlington
>>
>> IT Technical Assistant
>>
>>  
>>