[Evergreen-admin] slow(!) import

[Elliot Voris]

Meant for this to be a reply-all. Sorry!

Hey, Joe!

Welcome to Evergreen!! I think you'll really enjoy the community involvement and interaction. I've been a part of the lists long enough to know that this one doesn't usually see a lot of traffic. You'll probably be better suited sending this to the Evergreen Development list<mailto:open-ils-dev at list.georgialibraries.org> (don't forget to subscribe first<http://libmail.georgialibraries.org/mailman/listinfo/open-ils-dev>).

That's where I see most of the questions/answers take place. IRC is always a good option<http://evergreen-ils.org/irc.php>, too.

Welcome again! Good luck!

Thanks!

Elliot J. Voris

From: evergreen-admin-bounces at list.evergreen-ils.org [mailto:evergreen-admin-bounces at list.evergreen-ils.org] On Behalf Of Joe Thornton
Sent: 04 June 2013 13:37
To: evergreen-admin at list.evergreen-ils.org
Subject: [Evergreen-admin] slow(!) import

I'm new to Evergreen and to this list so I apologize in advance if this issue has been discussed already (I did look).

I installed Evergreen successfully on a test server with 16GB RAM and about 200GB of disk -- in two partitions.

We have:

Debian 7
Postgres 9.1 (not on a remote server)
Evergreen 2.4

To migrate bib records from our SirsiDynix Horizon database I used this document: http://docs.evergreen-ils.org/2.4/_migrating_your_bibliographic_records.html

The process was interrupted a few times by serious errors, but eventually I ended up with 550k bib records in the staging_records_import table.

The real problems started when I ran SELECT staging_importer();

The first time it stopped after many hours because it ran out of disk space. Postgres was using the smaller partition for data so I changed it to use the larger partition (~135GB) and restarted the job. This time it ran over the weekend and then ran out of disk space again.

Although this seems very strange to me, I started it again and this time the staging_records_import table has about 160k records in it.

I started SELECT staging_importer(); yesterday (about 24 hours ago) and it's still running and has used more than 50GB of disk so far.

Am I missing a step (or steps), or is this normal?

Thanks,

Joe Thornton
Manager, Automation Services
Upper Hudson Library System
28 Essex Street
Albany, NY 12206
518-437-9880 x230

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-admin/attachments/20130604/1311445b/attachment.html>