[OPEN-ILS-DEV] 3.0.0-betat1 database upgrade and reingest

[Mike Rylander]

(Excuse the brevity, on my phone...)

The ingest function did change, specifically to add display_field support.
The parameter order probably changed, as the intent is to keep the most
commonly used ones at the front of the default list.  pingest.pl will need
to account for the version now, or maybe it can use named parameter syntax.

https://www.postgresql.org/docs/9.4/static/sql-syntax-calling-funcs.html

Regards,

Mike Rylander

--   Sent from my phone, please pardon my thumbs.
 | President
 | Equinox Open Library Initiative
 | phone:  1-877-OPEN-ILS (673-6457)
 | email:  miker at EquinoxInitiative.org
 | web:  http://EquinoxInitiative.org

On Sep 21, 2017 7:43 PM, "Jason Stephenson" <jason at sigio.com> wrote:

> On 09/21/2017 03:32 PM, Josh Stompro wrote:
> > After the upgrade I started a reingest using the pingest.pl tool, since
> > I skipped all the skippable reingests during the upgrade, and I received
> > two instances of the following error.  I’m wondering if this is anything
> > to worry about?  Or is this just a timing issue, maybe two threads of
> > pingest.pl were both trying to create the same browse creator entry?  I
> > used small batches of 250 records, so maybe that is it?
> >
> >
> >
> > DBD::Pg::st execute failed: ERROR:  duplicate key value violates unique
> > constraint "browse_entry_sort_value_value_key"
> >
> > DETAIL:  Key (sort_value, value)=(glasrud clarence a creator, Glasrud,
> > Clarence A. creator) already exists.
> >
> > CONTEXT:  SQL statement "INSERT INTO metabib.browse_entry
> >
> >                     ( value, sort_value ) VALUES
> >
> >                     ( value_prepped, ind_data.sort_value )"
> >
> > PL/pgSQL function
> > metabib.reingest_metabib_field_entries(bigint,boolean,
> boolean,boolean,boolean,integer[])
> > line 86 at SQL statement at ./pingest.pl line 272.
> >
> > metabib.reingest_metabib_field_entries failed for record 8544 at
> > ./pingest.pl line 275.
>
> I'm curious to know what options you used when you ran pingest.pl and if
> it is actually a "stock" version of pingest.pl. That above error should
> not happen with the call on line 272. It deliberately skips doing the
> browse ingest.
>
> That said, I have not run pingest on a 3.0 database so something may
> have changed in the reingest_metabib_field_entries function that leads
> to this error.
>
> > I re-ran the pingest just for the records mentioned and that did take
> > care of adding the correct entries to metabib.browse_entry_def_map.
> >
> >
> >
> > Does each batch in pingest.pl run in its own transaction?  Would this
> > error mean that all the changes for that batch of 250 records were
> > rolled back?
>
> pingest.pl doesn't use transactions, so only the failed records need to
> be run again.
>
> pingest.pl splits the input into batches for the search, facets, and
> record attribute ingest which can be done in parallel. The browse ingest
> is done in one giant batch of all the records because of errors similar
> to the one you cite above when two batches try to update a record on the
> same authority.
>
> Like I said, I'm curious what options you used and what has changed in
> Evergreen 3.0.
>
> >
> >
> >
> > Thanks
> >
> >
> >
> > Lake Agassiz Regional Library - Moorhead MN larl.org
> >
> > Josh Stompro     | Office 218.233.3757 EXT-139
> >
> > LARL IT Director | Cell 218.790.2110
> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20170922/6e4b280c/attachment-0001.html>