[Evergreen-admin] slow(!) import

[Joe Thornton]

I'm new to Evergreen and to this list so I apologize in advance if this
issue has been discussed already (I did look).

I installed Evergreen successfully on a test server with 16GB RAM and about
200GB of disk -- in two partitions.

We have:

Debian 7
Postgres 9.1 (not on a remote server)
Evergreen 2.4

To migrate bib records from our SirsiDynix Horizon database I used this
document:
http://docs.evergreen-ils.org/2.4/_migrating_your_bibliographic_records.html

The process was interrupted a few times by serious errors, but eventually I
ended up with 550k bib records in the staging_records_import table.

The real problems started when I ran SELECT staging_importer();

The first time it stopped after many hours because it ran out of disk
space. Postgres was using the smaller partition for data so I changed it to
use the larger partition (~135GB) and restarted the job. This time it ran
over the weekend and then ran out of disk space again.

Although this seems very strange to me, I started it again and this time
the staging_records_import table has about 160k records in it.

I started SELECT staging_importer(); yesterday (about 24 hours ago) and
it's still running and has used more than 50GB of disk so far.

Am I missing a step (or steps), or is this normal?

Thanks,

Joe Thornton
Manager, Automation Services
Upper Hudson Library System
28 Essex Street
Albany, NY 12206
518-437-9880 x230
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-admin/attachments/20130604/e31f66b9/attachment.html>