[OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Thu May 22 18:21:41 EDT 2014

In the default configuration, Evergreen logs Apache errors via syslog, not the Apache error log.  You can either figure out where your syslog is logging to (sometime defaults to /var/log/syslog), or you can adjust your Apache config to use the error log you are used to.  Once you find the error message, feel free to post it if it doesn't make sense to you.

Thanks,
Dan

Daniel Wells
Library Programmer/Analyst
Hekman Library, Calvin College
616.526.7133

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Jesse McCarty
Sent: Thursday, May 22, 2014 11:17 AM
To: 'Evergreen Development Discussion List'
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

You have been amazing, thank you! The modified script ran with no issues after issuing the INSERT INTO command suggested. I only needed to modify one additional script (2.5.0-2.5.1-upgrade-db.sql) in order to complete the DB upgrade process. 

It looks like I am almost there with a complete upgrade. After running all reingest scripts, finishing the last few upgrade steps and starting all the services I can successfully connect via the staff client (either directly to the test server IP or the subdomain.domain.org format) and search patrons, books etc... with no apparent issues. From the web though, I can view the initial web landing page for our individual branches, login to my account and look through account settings. As soon as I try and perform an actual search, I get a 500 Internal Server Error (Screen shot attached)

So far I haven't been able to find anything in any of the logs that would indicate the cause and the error log for Apache is empty...

Thanks again, the assistance is greatly appreciated.

Jesse McCarty
City of Burlington
IT Technical Assistant

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Dan Wells
Sent: Wednesday, May 21, 2014 10:50 AM
To: Evergreen Development Discussion List
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Huh.  It's hard for me to guess how you might be missing that, but that's something which can be sorted out later if needed.  In the meantime, you'll want to recreate it with something like this:

INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath) VALUES
    (30, 'identifier', 'lccn', 'LC Control Number', 'marcxml', $$//marc:datafield[@tag='010']/marc:subfield[@code="a" or @code='z']$$);

We won't worry about actually populating the metabib.identifier_field_entry table, since you'll probably end up reingesting for 2.5 anyway.  Otherwise, a quick and dirty population would be something like:

INSERT INTO metabib.identifier_field_entry(source, field, value)
    SELECT record, 30, value FROM metabib.full_rec WHERE tag = '010' AND subfield IN ('a', 'z');

Let us know how it goes.

Thanks,
Dan

Daniel Wells
Library Programmer/Analyst
Hekman Library, Calvin College
616.526.7133

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Jesse McCarty
Sent: Wednesday, May 21, 2014 1:10 PM
To: 'Evergreen Development Discussion List'
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Thanks again for the great information. This particular upgrade has been quite the learning experience (Thank God for Virtual Test servers and snapshots!) and much more involved than the 2.4.0-2.4.4 upgrade I performed and the 2.4.4.-2.4.7 test upgrade.

To run the commands listed, I opened a PostgreSQL terminal (typing psql from the command line) and then ran the command: SELECT id FROM config.metabib_field WHERE field_class = 'identifier' AND name = 'lccn';

This resulted in the following output:

id
----
(0 rows)

Thanks again, the help is very much appreciated.

Jesse McCarty
City of Burlington
IT Technical Assistant

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Dan Wells
Sent: Wednesday, May 21, 2014 9:27 AM
To: Evergreen Development Discussion List
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Hello Jesse,

There isn't really such a thing as a 'fresh install' when talking about the DB side.  But yes, DB upgrades that fall outside the one supported path are essentially expected to be handled in a custom, case-by-case fashion.  Another factor to consider is that the farther you upgrade from the 'expected' upgrade path, the more likely you will have additional conflicts in even the smaller point-release scripts.  For instance, if you are at 2.4.7, you would likely have conflicts with not only the 2.4.3-2.5.0 script, but also the 2.5.0-2.5.1 or the 2.5.1-2.5.2 scripts, etc.  It's all going to depend on timing of the various fixes, and which version of each line they made it into.

As I said before, this is a known issue, and the most likely solution will be getting rid of the "packaged" scripts entirely, and having each upgrade be smart enough to check for and apply the individual upgrades it requires.  Many of the pieces to support this workflow are already in place, and it's mostly a matter of finding time to work out the details.

Now, for your specific upgrade issue, you're install does not have the "lccn" metabib_field entry in the expected place (id 30).  I'm not sure why this is, but the history of the table is a little messy due to us not originally reserving the lower IDs for system use.  This was corrected at some point in a "best effort" fashion, but there are many cases where these entries could be out of place.  Since it appears you have nothing with id = 30, your best bet is to find your 'lccn' entry (assuming it exists):

SELECT id FROM config.metabib_field WHERE field_class = 'identifier' AND name = 'lccn';

then update it back to 30:

BEGIN;
UPDATE config.metabib_field SET id = 30 WHERE id = [id you found, no brackets]; UPDATE metabib.identifier_field_entry SET field = 30 WHERE field = [id you found, no brackets]; COMMIT;

I think any other key relationships will cascade, so if the above works, then your 2.4.4-2.5.0 script should now work as well.

Good luck,
Dan

Daniel Wells
Library Programmer/Analyst
Hekman Library, Calvin College
616.526.7133

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Jesse McCarty
Sent: Wednesday, May 21, 2014 11:09 AM
To: 'Evergreen Development Discussion List'
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Thanks for the additional information Dan. Is this going to carry over when trying to upgrade to the 2.6 branch (or 2.7 etc...) as well? Seems like everyone on 2.4.4+ would be stuck needing to do a fresh install to upgrade then?

I have modified the 2.4.3-2.5.0-upgrade-db.sql script several times trying to flush out all the errors and get a clean run to proceed with the next step, but continue to have issues come up and am at a loss of where to go next (I do have VM snapshots to revert to and try again). So far where I am now is detailed below with attached log files and modified upgrade-db.sql file for reference.

After running into the initial error I commented out lines #592 & 593 and re-ran the script using the command "psql -U evergreen -f 2.4.3-2.5.0-upgrade-db.sql > /tmp/log.apply243-250_2 2>&1 &" (command provided by Galen in a previous response). The resulting file (log.apply243-250_2.rtf) is attached with an added .rtf file extension. This produced more errors in the file so next I commented out lines #12126 through #12133 re-ran script that produced the results shown in log.apply243-250_3 (also attached as an .rtf file)

After more errors, I commented out lines #12148 through #12167 re-ran the script which produced the results detailed in log.apply243-250_4.rtf

Again, more errors. This time commenting out lines #15791 through #15869 and then ran the script again producing log.apply243-250_5.rtf

After running and seeing still more errors, I commented out lines #15872 through #15961 to produce log.apply243-250_6.rtf (Screenshot of the error portion also attached).

I am stuck here, the other error messages gave an error where I could comment out that part of the script where the latest error has to deal with table modifications. I have also attached the 2.4.3-2.5.0-upgrade-db.sql file (renamed to 2.4.4-2.5.0-upgrade-db.sql) I have modified during this process.

Thanks for all the help.

Jesse McCarty
City of Burlington
IT Technical Assistant

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Dan Wells
Sent: Tuesday, May 20, 2014 10:56 AM
To: Evergreen Development Discussion List
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Hello Jesse,

You want to comment everything from the failing "SELECT evergreen.upgrade_deps_block_check..." up to (but not including) the next " SELECT evergreen.upgrade_deps_block_check...".  You may need to do this several times before you find them all.

Those checks exist precisely to cause the upgrade to fail if a piece of the script has already been applied.  Commenting out only these check lines *might* work (depending on the content of that section), but is certainly not recommended.

Also, the root of this problem is that there is no supported way to upgrade from 2.4.4+ to 2.5.0.  This is a known issue with no simple fix, other than to say the 2.x maintainer needs to make a new upgrade script for every 2.(x-1) release.  I am not outright opposed to that, but up to this point nobody has argued that the benefits would justify the cost.

Thanks,
Dan

Daniel Wells
Library Programmer/Analyst
Hekman Library, Calvin College
616.526.7133

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Jesse McCarty
Sent: Tuesday, May 20, 2014 11:19 AM
To: 'Evergreen Development Discussion List'
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Thanks Martha,

How much is there to comment out? Is it just the single line(s) that reads:

SELECT evergreen.upgrade_deps_block_check('0841', :eg_version); SELECT evergreen.upgrade_deps_block_check('0842', :eg_version);

Thanks again,

Jesse McCarty
City of Burlington
IT Technical Assistant

-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Martha Driscoll
Sent: Monday, May 19, 2014 2:30 PM
To: open-ils-dev at list.georgialibraries.org
Subject: Re: [OPEN-ILS-DEV] Errors Upgrading Evergreen 2.4.4 to 2.5.4

Jesse,
I found that upgrade scripts 0841 and 0842 which are contained in the
2.4.3-2.5.0 upgrade script were already applied in the upgrade to 2.4.4. 
  I commented out those two parts of the upgrade script.

Martha Driscoll
Systems Manager
North of Boston Library Exchange
Danvers, Massachusetts
www.noblenet.org

On 5/19/2014 1:31 PM, Jesse McCarty wrote:
> Hi,
>
>
> I am going through the process of upgrading our 2.4.4 installation of 
> Evergreen to 2.5.4. Currently running through the install on a test 
> server (which is an exact copy of our production server as it sat 
> several weeks ago, albeit with a different IP Address).
>
> After upgrading OpenSRF to 2.2.2 and the Evergreen code, I started 
> getting errors when running the update DB scripts. I started by 
> running the 2.4.3-2.5.0-upgrade-db.sql (There was no
> 2.4.4-2.5.0-upgrade-db.sql) with more errors that could fit on a 
> screen shot, then ran into additional errors on different update scripts:
>
> For 2.4.3-2.5.0-upgrade-db.sql errors see attached 243-250Error.JPG 
> (this also shows the reingest records script information, which I ran 
> after running all the DB upgrade scripts).
>
> For 2.5.0-2.5.1-upgrade-db.sql errors see attached 250-251Error.JPG
>
> No errors where reported when running the 2.5.1-2.5.2-upgrade-db.sql script.
>
> For 2.5.2-2.5.3-upgrade-db.sql errors see attached 252-253Error.JPG
>
> For 2.5.3-2.5.4-upgrade-db.sql errors see attached 253-254Error.JPG
>
>
> After starting Evergreen, I can connect via the web browser and login 
> to my account with no issues, searching the catalog produces and 
> internal server error. I can also connect to the staff client and 
> register the workstation and test/add SSL exemption with no issues.
> Once the work station is registered I get a Network error (can ping 
> the server from the workstation connecting) show in the attached 
> StaffClientError.JPG file. If I close that error the Staff Client 
> loads and then I can browse patrons and interact with the Server OK.
>
> I have a snapshot of the VM I can revert to that was taken prior to 
> any update scripts running against the DB to work through the process 
> again for further testing if needed.
>
> Thanks,
>
>
> Jesse McCarty
>
> City of Burlington
>
> IT Technical Assistant
>