[OPEN-ILS-GENERAL] ***SPAM*** Re: ***SPAM*** Importing bib and holdings data into Evergreen 2.4.2

[Václav Jansa]

Hello Chris,
i reply as a IT part of our Green home team;-)

Its clean EG 2.4.2 installation on 64bit debian-wheezy, virtual machine 
use 10GB image
free space is
opensrf at evergree-nb:~/KKB$ df -h
Filesystem                                              Size  Used Avail 
Use% Mounted on
rootfs                                                  9.6G 2.5G  6.6G  
28% /
udev                                                     10M 0   10M   
0% /dev
tmpfs                                                   202M 216K  
202M   1% /run
/dev/disk/by-uuid/e1cc81ee-2958-4d5d-941f-eab6798efe35  9.6G 2.5G  6.6G  
28% /
tmpfs                                                   5.0M 0  5.0M   
0% /run/lock
tmpfs                                                   497M 0  497M   
0% /run/shm

Old migration steps (from ESI migration tools) work for bibliographic 
records almost fine. Cca 50 mins for full import (from start SQL 
injection to searching in staff OPAC).

It's my second migration to EG (previous in EG 2.0 times).

Thanks for advance

Vaclav Jansa



On 10/15/2013 12:00 PM, Sharp, Chris wrote:
> Hi Linda,
>
> This appears to be the problem:
>
> ERROR:  could not extend file "base/42287/42883": No space left on device
> HINT:  Check free disk space.
>
> You can verify disk space with the bash command 'df -h', but it looks like you need to increase your disk space before continuing.
>
> Note that this is a PostgreSQL-level issue and is not specific to Evergreen.
>
> Hope that's helpful!
>
> Chris
>
> ----- Original Message -----
>> From: "Linda Jansova" <skolkova at chello.cz>
>> To: "Evergreen Discussion Group" <open-ils-general at list.georgialibraries.org>
>> Sent: Tuesday, October 15, 2013 4:33:28 AM
>> Subject: [OPEN-ILS-GENERAL] ***SPAM*** Importing bib and holdings data into	Evergreen 2.4.2
>>
>>
>> Dear all,
>>
>> We have attempted to import bibliographic and holdings data into
>> Evergreen 2.4.2 using instructions from chapter 14 (
>> http://docs.evergreen-ils.org/2.4/_migrating_from_a_legacy_system.html
>> ) of the official Evergreen documentation.
>>
>> First, we were unsuccessful in using the create function (step 3):
>>
>> ***
>> CREATE OR REPLACE FUNCTION staging_importer() RETURNS NULL AS $$
>> DECLARE stage RECORD;
>> BEGIN
>> FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
>>        INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
>>        (stage.marc, 'IMPORT');
>>        UPDATE staging_records_import SET dest =
>>        currval('biblio.record_entry_id_seq');
>>     END LOOP;
>>    END;
>>    $$ LANGUAGE plpgsql;
>>
>>
>> ***
>>
>> However, it worked okay when we substituted NULL for VOID as
>> recommended at
>> http://comments.gmane.org/gmane.education.libraries.open-ils.general/7820
>> . Then it took well over 12 hours to import 9460 records. Yet, the
>> import was unsuccessful:
>>
>> ***
>> evergreen=# CREATE TABLE staging_records_import (id BIGSERIAL, dest
>> BIGINT, marc TEXT);
>>
>> evergreen=# CREATE OR REPLACE FUNCTION staging_importer() RETURNS
>> VOID AS $$
>> DECLARE stage RECORD;
>> BEGIN
>> FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
>> INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
>> (stage.marc, 'IMPORT');
>> UPDATE staging_records_import SET dest =
>> currval('biblio.record_entry_id_seq');
>> END LOOP;
>> END;
>> $$ LANGUAGE plpgsql;
>> CREATE FUNCTION
>>
>>
>> evergreen=# COPY staging_records_import (marc) FROM
>> '/home/opensrf/KKB/IMPORT.xml';
>> COPY 9460
>>
>>
>> evergreen=# SELECT staging_importer();
>> ERROR: could not extend file "base/42287/42883": No space left on
>> device
>> HINT: Check free disk space.
>> CONTEXT: SQL statement "INSERT INTO metabib.real_full_rec (record,
>> tag, ind1, ind2, subfield, value)
>> SELECT record, tag, ind1, ind2, subfield, value FROM
>> biblio.flatten_marc( bib_id )"
>> PL/pgSQL function "reingest_metabib_full_rec" line 7 at SQL statement
>> SQL statement "SELECT metabib.reingest_metabib_full_rec(NEW.id)"
>> PL/pgSQL function "indexing_ingest_or_delete" line 46 at PERFORM
>> SQL statement "INSERT INTO biblio.record_entry (marc, last_xact_id)
>> VALUES (stage.marc, 'IMPORT')"
>> PL/pgSQL function "staging_importer" line 5 at SQL statement
>>
>> ***
>>
>> The statistics of the virtual machine load have showed that the in
>> the first hour disk I/O write operation (at about 30 MBps)
>> prevailed, while in the rest of the time disk I/O read operations
>> (at about 70 MBps) were prevalent.
>>
>> When we tried to use the import method recommended for previous
>> versions (especially version 2.1:
>> http://docs.evergreen-ils.org/2.1/html/migrating_records_using_migration_tools.html
>> ), the import of bibliographic data was successful (and it took only
>> about an hour overall). However, when we subsequently wanted to
>> import holdings data, the system was not able to fill the column
>> entitled egid and therefore we could not import holdings data.
>>
>> Any ideas how to sort this out? Or which import methods to use for
>> Evergreen 2.4.2?
>>
>> Thanks in advance for any hints!
>>
>> Linda and Vaclav Jansovi :-)
>>
>>
>>