[OPEN-ILS-DEV] 3.0.0-betat1 database upgrade and reingest

[Jason Stephenson]

On 09/21/2017 03:32 PM, Josh Stompro wrote:
> After the upgrade I started a reingest using the pingest.pl tool, since
> I skipped all the skippable reingests during the upgrade, and I received
> two instances of the following error.  I’m wondering if this is anything
> to worry about?  Or is this just a timing issue, maybe two threads of
> pingest.pl were both trying to create the same browse creator entry?  I
> used small batches of 250 records, so maybe that is it? 
> 
>  
> 
> DBD::Pg::st execute failed: ERROR:  duplicate key value violates unique
> constraint "browse_entry_sort_value_value_key"
> 
> DETAIL:  Key (sort_value, value)=(glasrud clarence a creator, Glasrud,
> Clarence A. creator) already exists.
> 
> CONTEXT:  SQL statement "INSERT INTO metabib.browse_entry
> 
>                     ( value, sort_value ) VALUES
> 
>                     ( value_prepped, ind_data.sort_value )"
> 
> PL/pgSQL function
> metabib.reingest_metabib_field_entries(bigint,boolean,boolean,boolean,boolean,integer[])
> line 86 at SQL statement at ./pingest.pl line 272.
> 
> metabib.reingest_metabib_field_entries failed for record 8544 at
> ./pingest.pl line 275.

I'm curious to know what options you used when you ran pingest.pl and if
it is actually a "stock" version of pingest.pl. That above error should
not happen with the call on line 272. It deliberately skips doing the
browse ingest.

That said, I have not run pingest on a 3.0 database so something may
have changed in the reingest_metabib_field_entries function that leads
to this error.

> I re-ran the pingest just for the records mentioned and that did take
> care of adding the correct entries to metabib.browse_entry_def_map.
> 
>  
> 
> Does each batch in pingest.pl run in its own transaction?  Would this
> error mean that all the changes for that batch of 250 records were
> rolled back?

pingest.pl doesn't use transactions, so only the failed records need to
be run again.

pingest.pl splits the input into batches for the search, facets, and
record attribute ingest which can be done in parallel. The browse ingest
is done in one giant batch of all the records because of errors similar
to the one you cite above when two batches try to update a record on the
same authority.

Like I said, I'm curious what options you used and what has changed in
Evergreen 3.0.

> 
>  
> 
> Thanks
> 
>  
> 
> Lake Agassiz Regional Library - Moorhead MN larl.org
> 
> Josh Stompro     | Office 218.233.3757 EXT-139
> 
> LARL IT Director | Cell 218.790.2110 
> 
>  
>