[OPEN-ILS-DEV] slow(!) import

[Jason Stephenson]

Quoting Joe Thornton <jthornton at uhls.lib.ny.us>:

> I'm new to Evergreen and to this list so I apologize in advance if this
> issue has been discussed already (I did look).
>
> I installed Evergreen successfully on a test server with 16GB RAM and about
> 200GB of disk -- in two partitions.

Is this for production use or are you just kicking the tires at this point?

>
> We have:
>
> Debian 7
> Postgres 9.1 (not on a remote server)
> Evergreen 2.4
>
> To migrate bib records from our SirsiDynix Horizon database I used this
> document:
> http://docs.evergreen-ils.org/2.4/_migrating_your_bibliographic_records.html

I migrated our Horizon system to Evergreen in 2011. Here is the  
software that resulted:  
http://git.mvlcstaff.org/?p=jason/hz2eg.git;a=summary

You might find it useful, you might not, but it is there.


>
> The process was interrupted a few times by serious errors, but eventually I
> ended up with 550k bib records in the staging_records_import table.
>
> The real problems started when I ran SELECT staging_importer();
>
> The first time it stopped after many hours because it ran out of disk
> space. Postgres was using the smaller partition for data so I changed it to
> use the larger partition (~135GB) and restarted the job. This time it ran
> over the weekend and then ran out of disk space again.
>
> Although this seems very strange to me, I started it again and this time
> the staging_records_import table has about 160k records in it.
>
> I started SELECT staging_importer(); yesterday (about 24 hours ago) and
> it's still running and has used more than 50GB of disk so far.
>
> Am I missing a step (or steps), or is this normal?

I've done/helped with a few migrations and to be honest, I've not had  
much success using the stock import tools. I generally do things  
differently.

I do suspect, however, that you are going to need more disk space for  
your database. If you intend to use this in production, you will  
definitely want a more powerful database server that is on separate  
hardware from that which runs your Evergreen services.


>
> Thanks,
>
> Joe Thornton
> Manager, Automation Services
> Upper Hudson Library System
> 28 Essex Street
> Albany, NY 12206
> 518-437-9880 x230
>



-- 
Jason Stephenson
Assistant Director for Technology Services
Merrimack Valley Library Consortium