[OPEN-ILS-DEV] broken private router connection - database issue

[Dan Scott]

On Mon, May 16, 2011 at 03:53:03PM +0200, Daniël Kollmer wrote:
> Hello all
> 
> We are running Evergreen 2.0.3 at the moment, and after we have imported our whole production catalogue, we were hitting the limits of our storage capacity. Slightly surprised about the size of the database (something which took 4GB of db records on our old system became close to 25GB in the evergreen postgres db), we decided to use the postgres vacuum function. Indeed we saw that a lot of redundancy seemed to be removed.

"Hitting the limits of storage capacity" doesn't sound good; I've seen
the VACUUM operation take frightening amounts of disk space to complete,
but it seems to respect when there are only a few percent of disk space.
In our case, I saw 100 GB of disk space being consumed when VACUUM
was running full tilt.

> However, the vacuum operation broke off, and now we can not use Evergreen anymore. The logs show nothing (even on loglevel 4) the settings_tester.pl script says everything is fine. But still, if I try to log in to the srfsh, it works and I can send queries to the opensrf facilities. If I try logging in the staffclient, I get an error message though, and if I run autogen.sh I get the following:
> 
> Updating fieldmapper
> Updating web_fieldmapper
> Updating OrgTree
> Exception: OpenSRF::EX::Session 2011-05-16T15:47:48 OpenSRF::Transport /usr/local/share/perl/5.10.1/OpenSRF/Transport.pm:92 Session Error: router at private.evergreen2.iisg.nl/open-ils.cstore IS NOT CONNECTED TO THE NETWORK!!!
> 
> I have looked through the mailinglist archives and found a few similar cases where it was down to changed passwords in the router configuration. Those were not changed at all in our case though. Also, if 
> 
> Also, I have a clean database (without records) and if I use that one, it works
>
> One additional piece of information that might be relevant: I run the database with the full catalogue on an external host, but the clean one runs on localhost.

Hmm. If open-ils.cstore is dying, it sounds like your database might be
the problem. Can you connect to your database with the full catalogue
and select rows from major tables such as biblio.record_entry,
biblio.full_rec, asset.copy, asset.call_number, actor.usr,
actor.org_unit?

> Does anyone have an idea whether I could fix this? Otherwise we would have to do the whole import of 1.5mill records again

I don't suppose you have a backup of your database to restore from...

> On a sidenote, can someone tell me whether it is normal that a catalogue database with 1-1.5 mill records takes up 25GB?

I would say that's normal, based on my own experience.