[OPEN-ILS-GENERAL] Importing bib and holdings data into Evergreen 2.4.2
Václav Jansa
vaclav.jansa at gmail.com
Wed Oct 16 09:11:48 EDT 2013
Hi Chris,
i finished today morning new testing server hosted on my home
server/desktop/router/virthost...
New testing virtual server has this configuration 4GB RAM, 2core, 50GB HDD
64 bit Debian 7, EG 2.4.2
Import currently running over seven hours. 6k5 bib records (with minimal
record complexity) .. DB now consume 6GB HDD space.
Our production server 32bit Debian 6 EG 2.2.2 consume for DB 0.9GB HDD
space (35k bib records, 2 years of history).
Unfortunately i have no information about disk usage from migration to
2.4.2 with use of old migration steps from ESI migration... 6.5k records
has imported to virtual machine hosted on my laptop under 1 hour.
How much clean or dirty is usage of old ESI scripts and importing steps
on new EG (2.4.2) for bib records.
Thanks for advance
Vaclav
On 10/15/2013 01:23 PM, Sharp, Chris wrote:
> Hi Vaclav,
>
> 6.6 GB sounds a bit tight to me and it appears that it's not enough to accommodate the import Linda is attempting. Any chance at increasing that size? I think it will be necessary based on the error Linda shared in her post.
>
> Hope that's helpful,
>
> Chris
>
> ----- Original Message -----
>> From: "Václav Jansa" <vaclav.jansa at gmail.com>
>> To: open-ils-general at list.georgialibraries.org
>> Sent: Tuesday, October 15, 2013 6:16:27 AM
>> Subject: [OPEN-ILS-GENERAL] ***SPAM*** Re: ***SPAM*** Importing bib and holdings data into Evergreen 2.4.2
>>
>> Hello Chris,
>> i reply as a IT part of our Green home team;-)
>>
>> Its clean EG 2.4.2 installation on 64bit debian-wheezy, virtual
>> machine
>> use 10GB image
>> free space is
>> opensrf at evergree-nb:~/KKB$ df -h
>> Filesystem Size Used
>> Avail
>> Use% Mounted on
>> rootfs 9.6G 2.5G
>> 6.6G
>> 28% /
>> udev 10M 0 10M
>> 0% /dev
>> tmpfs 202M 216K
>> 202M 1% /run
>> /dev/disk/by-uuid/e1cc81ee-2958-4d5d-941f-eab6798efe35 9.6G 2.5G
>> 6.6G
>> 28% /
>> tmpfs 5.0M 0 5.0M
>> 0% /run/lock
>> tmpfs 497M 0 497M
>> 0% /run/shm
>>
>> Old migration steps (from ESI migration tools) work for bibliographic
>> records almost fine. Cca 50 mins for full import (from start SQL
>> injection to searching in staff OPAC).
>>
>> It's my second migration to EG (previous in EG 2.0 times).
>>
>> Thanks for advance
>>
>> Vaclav Jansa
>>
>>
>>
>> On 10/15/2013 12:00 PM, Sharp, Chris wrote:
>>> Hi Linda,
>>>
>>> This appears to be the problem:
>>>
>>> ERROR: could not extend file "base/42287/42883": No space left on
>>> device
>>> HINT: Check free disk space.
>>>
>>> You can verify disk space with the bash command 'df -h', but it
>>> looks like you need to increase your disk space before continuing.
>>>
>>> Note that this is a PostgreSQL-level issue and is not specific to
>>> Evergreen.
>>>
>>> Hope that's helpful!
>>>
>>> Chris
>>>
>>> ----- Original Message -----
>>>> From: "Linda Jansova" <skolkova at chello.cz>
>>>> To: "Evergreen Discussion Group"
>>>> <open-ils-general at list.georgialibraries.org>
>>>> Sent: Tuesday, October 15, 2013 4:33:28 AM
>>>> Subject: [OPEN-ILS-GENERAL] ***SPAM*** Importing bib and holdings
>>>> data into Evergreen 2.4.2
>>>>
>>>>
>>>> Dear all,
>>>>
>>>> We have attempted to import bibliographic and holdings data into
>>>> Evergreen 2.4.2 using instructions from chapter 14 (
>>>> http://docs.evergreen-ils.org/2.4/_migrating_from_a_legacy_system.html
>>>> ) of the official Evergreen documentation.
>>>>
>>>> First, we were unsuccessful in using the create function (step 3):
>>>>
>>>> ***
>>>> CREATE OR REPLACE FUNCTION staging_importer() RETURNS NULL AS $$
>>>> DECLARE stage RECORD;
>>>> BEGIN
>>>> FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
>>>> INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
>>>> (stage.marc, 'IMPORT');
>>>> UPDATE staging_records_import SET dest =
>>>> currval('biblio.record_entry_id_seq');
>>>> END LOOP;
>>>> END;
>>>> $$ LANGUAGE plpgsql;
>>>>
>>>>
>>>> ***
>>>>
>>>> However, it worked okay when we substituted NULL for VOID as
>>>> recommended at
>>>> http://comments.gmane.org/gmane.education.libraries.open-ils.general/7820
>>>> . Then it took well over 12 hours to import 9460 records. Yet, the
>>>> import was unsuccessful:
>>>>
>>>> ***
>>>> evergreen=# CREATE TABLE staging_records_import (id BIGSERIAL,
>>>> dest
>>>> BIGINT, marc TEXT);
>>>>
>>>> evergreen=# CREATE OR REPLACE FUNCTION staging_importer() RETURNS
>>>> VOID AS $$
>>>> DECLARE stage RECORD;
>>>> BEGIN
>>>> FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
>>>> INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
>>>> (stage.marc, 'IMPORT');
>>>> UPDATE staging_records_import SET dest =
>>>> currval('biblio.record_entry_id_seq');
>>>> END LOOP;
>>>> END;
>>>> $$ LANGUAGE plpgsql;
>>>> CREATE FUNCTION
>>>>
>>>>
>>>> evergreen=# COPY staging_records_import (marc) FROM
>>>> '/home/opensrf/KKB/IMPORT.xml';
>>>> COPY 9460
>>>>
>>>>
>>>> evergreen=# SELECT staging_importer();
>>>> ERROR: could not extend file "base/42287/42883": No space left on
>>>> device
>>>> HINT: Check free disk space.
>>>> CONTEXT: SQL statement "INSERT INTO metabib.real_full_rec (record,
>>>> tag, ind1, ind2, subfield, value)
>>>> SELECT record, tag, ind1, ind2, subfield, value FROM
>>>> biblio.flatten_marc( bib_id )"
>>>> PL/pgSQL function "reingest_metabib_full_rec" line 7 at SQL
>>>> statement
>>>> SQL statement "SELECT metabib.reingest_metabib_full_rec(NEW.id)"
>>>> PL/pgSQL function "indexing_ingest_or_delete" line 46 at PERFORM
>>>> SQL statement "INSERT INTO biblio.record_entry (marc,
>>>> last_xact_id)
>>>> VALUES (stage.marc, 'IMPORT')"
>>>> PL/pgSQL function "staging_importer" line 5 at SQL statement
>>>>
>>>> ***
>>>>
>>>> The statistics of the virtual machine load have showed that the in
>>>> the first hour disk I/O write operation (at about 30 MBps)
>>>> prevailed, while in the rest of the time disk I/O read operations
>>>> (at about 70 MBps) were prevalent.
>>>>
>>>> When we tried to use the import method recommended for previous
>>>> versions (especially version 2.1:
>>>> http://docs.evergreen-ils.org/2.1/html/migrating_records_using_migration_tools.html
>>>> ), the import of bibliographic data was successful (and it took
>>>> only
>>>> about an hour overall). However, when we subsequently wanted to
>>>> import holdings data, the system was not able to fill the column
>>>> entitled egid and therefore we could not import holdings data.
>>>>
>>>> Any ideas how to sort this out? Or which import methods to use for
>>>> Evergreen 2.4.2?
>>>>
>>>> Thanks in advance for any hints!
>>>>
>>>> Linda and Vaclav Jansovi :-)
>>>>
>>>>
>>>>
>>
More information about the Open-ils-general
mailing list