[OPEN-ILS-GENERAL] Import issues

Thu Aug 13 13:14:04 EDT 2009

Hello Dan,

Thanks for your prompt response. I tried those options with marc2bre.pl and
now the 'marc2bre.pl' produced no error. But, the problem about duplication
still persists while executing the SQL. I rebuilt the database several
times, and tried 'marc2bre.pl' with all the suggested options like you have
suggested.

I performed the following commands during the import process and they
produced SQL for each MARC records:

#perl marc2bre.pl --db_user postgres --db_host localhost --db_pw evergreen
--db_name evergreen --encoding UTF8 --idfield 852 --idsubfield x --tcnfield
852 --tcnsubfield x 8500.mrc > 8500.bre
#perl direct_ingest.pl ~/8500.bre > ~/8500.ingest
#perl pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or msfe -or
mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a msefe
--output=8500.sql < ~/8500.ingest
#cp 8500.sql ~
#psql -U evergreen evergreen
evergreen=# \i ~/8500.sql

After rebuilding the database, any generated SQL is executed successfully.
But, after the first successful execution of any generated SQL, any other
generated SQL files are not executed. I've attached the generated SQL errors
herewith.

I look forward to hearing from you.

Thank you.

--
Dibyendra Hyoju
Madan Puraskar Pustakalaya
Patan Dhoka, Lalitpur
Nepal

On Thu, Aug 13, 2009 at 12:14 PM, Dan Scott <denials at gmail.com> wrote:

> Sorry Dibyendra, those options should have been --tcnfield and
> --tcnsubfield
>
> Try those, they should work in 1.4.
>
> Dan
>
> 2009/8/13 Dibyendra Hyoju <dibyendra at gmail.com>:
> > Hello Dan,
> >
> > Thank you very much for your response.
> >
> > I used the options: 'tcn_field' and 'tcn_subfield', but they are not
> > recognized by the marc2bre.pl. I executed the marc2bre.pl in following
> >  way and the output is as follows:
> >
> >
> dibyendra-laptop:/home/opensrf/Evergreen-ILS-1.4.0.4/Open-ILS/src/extras/import#
> > perl marc2bre.pl --db_user postgres --db_host localhost --db_pw
> > evergreen --db_name evergreen --encoding UTF8 --idfield 852
> > --idsubfield x --tcn_field 852 --tcn_subfield x 8500.mrc > 8500.bre
> > Unknown option: tcn_field
> > Unknown option: tcn_subfield
> >
> > I guess, these options: 'tcn_field' and 'tcn_subfield' are not
> > implemented on EG 1.4 yet. Is that so? I couldn't find EG 1.6 on the
> > download page. We're planning to migrate our 10,000 validated MARC
> > records into Evergreen.
> >
> > Thank you.
> >
> > --
> > Dibyendra Hyoju
> > Madan Puraskar Pustakalaya
> > Patan Dhoka, Lalipur
> > Nepal
> >
> > On Tue, Aug 11, 2009 at 10:15 AM, Dan Scott <denials at gmail.com> wrote:
> >>
> >> 2009/8/8 Dibyendra Hyoju <dibyendra at gmail.com>:
> >> > Hello all,
> >> > I imported the sample records again on the another machine having
> Evergreen
> >> > 1.4, and after importing that record, I couldn't import any records
> from
> >> > other MARC files. I have attached the record '8500.mrc' and '8400.mrc'
> >> > herewith. I first executed the SQL generated from 8500.mrc'
> successfully.
> >> > Then, I couldn't execute the SQL generated from '8400.mrc'. The error
> is
> >> > same as before like 'ERROR:  duplicate key violates unique constraint
> >> > "biblio_record_unique_tcn"'. I only used the option "--encoding UTF8"
> this
> >> > time. I tried few other records, but I got the same error. Few of the
> tested
> >> > records are attached herewith if somebody wants to volunteer the test.
> If
> >> > anyone has faced this problem before and have found the solution,
> please
> >> > help. Any help will be highly appreciated.
> >>
> >> Sorry for the delayed reply, I'm on leave at the moment and not
> >> connected very often.
> >>
> >> marc2bre.pl doesn't really deal with automatically generated TCNs all
> >> that well, so you're best off explicitly identifying a source for the
> >> TCN. To avoid getting duplicate TCN values and record ID values if all
> >> of your records follow the same pattern with the 852 $x field /
> >> subfield accession number identifier, you should also use the
> >> --tcn_field / --tcn_subfield options for marc2bre.pl:
> >>
> >> perl marc2bre.pl --db_user postgres --db_host localhost --db_pw
> >> evergreen --db_name evergreen --encoding UTF8 --idfield 852
> >> --idsubfield x --tcn_field 852 --tcn_subfield x 8400.mrc > 8400.bre
> >>
> >> I tried importing both your 8000.mrc and 8400.mrc files with the and
> >> these options worked fine for me with no duplicate value warnings.
> >> This is with Evergreen rel_1_6_0 on Ubuntu 8.04, but marc2bre.pl (the
> >> most critical script for parsing out TCN and record number) shows no
> >> significant differences between 1.4 and rel_1_6_0.
> >>
> >> --
> >> Dan Scott
> >> Laurentian University
> >
>
>
>
> --
> Dan Scott
> Laurentian University
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://libmail.georgialibraries.org/pipermail/open-ils-general/attachments/20090813/32cfffd6/attachment.htm 
-------------- next part --------------
evergreen=# \i ~/8500.sql
SET
BEGIN
psql:/root/8500.sql:103: ERROR:  duplicate key violates unique constraint "biblio_record_unique_tcn"
CONTEXT:  COPY record_entry, line 1: "t	now	1	f	now	1	svasnimancheba	8500	IMPORT-1250181903.85429<record    xmlns:xsi="http://www.w3.org/..."
psql:/root/8500.sql:105: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8500.sql:107: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8500.sql:108: invalid command \N
psql:/root/8500.sql:109: invalid command \N
psql:/root/8500.sql:110: invalid command \N
psql:/root/8500.sql:111: invalid command \N
psql:/root/8500.sql:112: invalid command \N
psql:/root/8500.sql:113: invalid command \N
psql:/root/8500.sql:114: invalid command \N
psql:/root/8500.sql:115: invalid command \N
psql:/root/8500.sql:116: invalid command \N
psql:/root/8500.sql:117: invalid command \N
psql:/root/8500.sql:118: invalid command \N
psql:/root/8500.sql:119: invalid command \N
psql:/root/8500.sql:120: invalid command \N
psql:/root/8500.sql:121: invalid command \N
psql:/root/8500.sql:122: invalid command \N
psql:/root/8500.sql:123: invalid command \N
psql:/root/8500.sql:124: invalid command \N
psql:/root/8500.sql:125: invalid command \N
psql:/root/8500.sql:126: invalid command \N
psql:/root/8500.sql:127: invalid command \N
psql:/root/8500.sql:128: invalid command \N
psql:/root/8500.sql:129: invalid command \N
psql:/root/8500.sql:130: invalid command \N
psql:/root/8500.sql:131: invalid command \N
psql:/root/8500.sql:132: invalid command \N
psql:/root/8500.sql:133: invalid command \N
psql:/root/8500.sql:134: invalid command \N
psql:/root/8500.sql:135: invalid command \N
psql:/root/8500.sql:136: invalid command \N
psql:/root/8500.sql:137: invalid command \N
psql:/root/8500.sql:138: invalid command \N
psql:/root/8500.sql:139: invalid command \N
psql:/root/8500.sql:140: invalid command \N
psql:/root/8500.sql:141: invalid command \N
psql:/root/8500.sql:142: invalid command \N
psql:/root/8500.sql:143: invalid command \N
psql:/root/8500.sql:144: invalid command \N
psql:/root/8500.sql:145: invalid command \N
psql:/root/8500.sql:146: invalid command \N
psql:/root/8500.sql:147: invalid command \N
psql:/root/8500.sql:148: invalid command \N
psql:/root/8500.sql:149: invalid command \N
psql:/root/8500.sql:150: invalid command \N
psql:/root/8500.sql:151: invalid command \N
psql:/root/8500.sql:152: invalid command \N
psql:/root/8500.sql:153: invalid command \N
psql:/root/8500.sql:154: invalid command \N
psql:/root/8500.sql:155: invalid command \N
psql:/root/8500.sql:156: invalid command \N
psql:/root/8500.sql:157: invalid command \N
psql:/root/8500.sql:158: invalid command \N
psql:/root/8500.sql:159: invalid command \N
psql:/root/8500.sql:160: invalid command \N
psql:/root/8500.sql:161: invalid command \N
psql:/root/8500.sql:162: invalid command \N
psql:/root/8500.sql:163: invalid command \N
psql:/root/8500.sql:164: invalid command \N
psql:/root/8500.sql:165: invalid command \N
psql:/root/8500.sql:166: invalid command \N
psql:/root/8500.sql:167: invalid command \N
psql:/root/8500.sql:168: invalid command \N
psql:/root/8500.sql:169: invalid command \N
psql:/root/8500.sql:170: invalid command \N
psql:/root/8500.sql:171: invalid command \N
psql:/root/8500.sql:172: invalid command \N
psql:/root/8500.sql:173: invalid command \N
psql:/root/8500.sql:174: invalid command \N
psql:/root/8500.sql:175: invalid command \N
psql:/root/8500.sql:176: invalid command \N
psql:/root/8500.sql:177: invalid command \N
psql:/root/8500.sql:178: invalid command \N
psql:/root/8500.sql:179: invalid command \N
psql:/root/8500.sql:180: invalid command \N
psql:/root/8500.sql:181: invalid command \N
psql:/root/8500.sql:182: invalid command \N
psql:/root/8500.sql:183: invalid command \N
psql:/root/8500.sql:184: invalid command \N
psql:/root/8500.sql:185: invalid command \N
psql:/root/8500.sql:186: invalid command \N
psql:/root/8500.sql:187: invalid command \N
psql:/root/8500.sql:188: invalid command \N
psql:/root/8500.sql:189: invalid command \N
psql:/root/8500.sql:190: invalid command \N
psql:/root/8500.sql:191: invalid command \N
psql:/root/8500.sql:192: invalid command \N
psql:/root/8500.sql:193: invalid command \N
psql:/root/8500.sql:194: invalid command \N
psql:/root/8500.sql:195: invalid command \N
psql:/root/8500.sql:196: invalid command \N
psql:/root/8500.sql:197: invalid command \N
psql:/root/8500.sql:198: invalid command \N
psql:/root/8500.sql:199: invalid command \N
psql:/root/8500.sql:200: invalid command \N
psql:/root/8500.sql:201: invalid command \N
psql:/root/8500.sql:202: invalid command \N
psql:/root/8500.sql:203: invalid command \N
psql:/root/8500.sql:204: invalid command \N
psql:/root/8500.sql:205: invalid command \.
psql:/root/8500.sql:208: ERROR:  syntax error at or near "m"
LINE 1: m a 
        ^
psql:/root/8500.sql:209: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8500.sql:210: invalid command \N
psql:/root/8500.sql:211: invalid command \N
psql:/root/8500.sql:212: invalid command \N
psql:/root/8500.sql:256: invalid command \N
psql:/root/8500.sql:257: invalid command \N
psql:/root/8500.sql:258: invalid command \N
psql:/root/8500.sql:304: invalid command \N
psql:/root/8500.sql:305: invalid command \N
psql:/root/8500.sql:306: invalid command \N
psql:/root/8500.sql:352: invalid command \N
psql:/root/8500.sql:353: invalid command \N
psql:/root/8500.sql:354: invalid command \N
psql:/root/8500.sql:400: invalid command \N
psql:/root/8500.sql:401: invalid command \N
psql:/root/8500.sql:402: invalid command \N
psql:/root/8500.sql:450: invalid command \N
psql:/root/8500.sql:451: invalid command \N
psql:/root/8500.sql:452: invalid command \N
psql:/root/8500.sql:499: invalid command \N
psql:/root/8500.sql:500: invalid command \N
psql:/root/8500.sql:501: invalid command \N
psql:/root/8500.sql:549: invalid command \N
psql:/root/8500.sql:550: invalid command \N
psql:/root/8500.sql:551: invalid command \N
psql:/root/8500.sql:599: invalid command \N
psql:/root/8500.sql:600: invalid command \N
psql:/root/8500.sql:601: invalid command \N
psql:/root/8500.sql:650: invalid command \N
psql:/root/8500.sql:651: invalid command \N
psql:/root/8500.sql:652: invalid command \N
psql:/root/8500.sql:704: invalid command \N
psql:/root/8500.sql:705: invalid command \N
psql:/root/8500.sql:706: invalid command \N
psql:/root/8500.sql:753: invalid command \N
psql:/root/8500.sql:754: invalid command \N
psql:/root/8500.sql:755: invalid command \N
psql:/root/8500.sql:795: invalid command \N
psql:/root/8500.sql:796: invalid command \N
psql:/root/8500.sql:797: invalid command \N
psql:/root/8500.sql:843: invalid command \N
psql:/root/8500.sql:844: invalid command \N
psql:/root/8500.sql:845: invalid command \N
psql:/root/8500.sql:893: invalid command \N
psql:/root/8500.sql:894: invalid command \N
psql:/root/8500.sql:895: invalid command \N
psql:/root/8500.sql:943: invalid command \N
psql:/root/8500.sql:944: invalid command \N
psql:/root/8500.sql:945: invalid command \N
psql:/root/8500.sql:993: invalid command \N
psql:/root/8500.sql:994: invalid command \N
psql:/root/8500.sql:995: invalid command \N
psql:/root/8500.sql:1043: invalid command \N
psql:/root/8500.sql:1044: invalid command \N
psql:/root/8500.sql:1045: invalid command \N
psql:/root/8500.sql:1090: invalid command \N
psql:/root/8500.sql:1091: invalid command \N
psql:/root/8500.sql:1092: invalid command \N
psql:/root/8500.sql:1141: invalid command \N
psql:/root/8500.sql:1142: invalid command \N
psql:/root/8500.sql:1143: invalid command \N
psql:/root/8500.sql:1188: invalid command \N
psql:/root/8500.sql:1189: invalid command \N
psql:/root/8500.sql:1190: invalid command \N
psql:/root/8500.sql:1237: invalid command \N
psql:/root/8500.sql:1238: invalid command \N
psql:/root/8500.sql:1239: invalid command \N
psql:/root/8500.sql:1290: invalid command \N
psql:/root/8500.sql:1291: invalid command \N
psql:/root/8500.sql:1292: invalid command \N
psql:/root/8500.sql:1342: invalid command \N
psql:/root/8500.sql:1343: invalid command \N
psql:/root/8500.sql:1344: invalid command \N
psql:/root/8500.sql:1388: invalid command \N
psql:/root/8500.sql:1389: invalid command \N
psql:/root/8500.sql:1390: invalid command \N
psql:/root/8500.sql:1436: invalid command \N
psql:/root/8500.sql:1437: invalid command \N
psql:/root/8500.sql:1438: invalid command \N
psql:/root/8500.sql:1486: invalid command \N
psql:/root/8500.sql:1487: invalid command \N
psql:/root/8500.sql:1488: invalid command \N
psql:/root/8500.sql:1536: invalid command \N
psql:/root/8500.sql:1537: invalid command \N
psql:/root/8500.sql:1538: invalid command \N
psql:/root/8500.sql:1582: invalid command \N
psql:/root/8500.sql:1583: invalid command \N
psql:/root/8500.sql:1584: invalid command \N
psql:/root/8500.sql:1626: invalid command \N
psql:/root/8500.sql:1627: invalid command \N
psql:/root/8500.sql:1628: invalid command \N
psql:/root/8500.sql:1683: invalid command \N
psql:/root/8500.sql:1684: invalid command \N
psql:/root/8500.sql:1685: invalid command \N
psql:/root/8500.sql:1728: invalid command \N
psql:/root/8500.sql:1729: invalid command \N
psql:/root/8500.sql:1730: invalid command \N
psql:/root/8500.sql:3427: invalid command \N
psql:/root/8500.sql:3428: invalid command \N
psql:/root/8500.sql:3429: invalid command \N
psql:/root/8500.sql:3487: invalid command \N
psql:/root/8500.sql:3488: invalid command \N
psql:/root/8500.sql:3489: invalid command \N
psql:/root/8500.sql:4132: invalid command \N
psql:/root/8500.sql:4133: invalid command \N
psql:/root/8500.sql:4134: invalid command \N
psql:/root/8500.sql:4184: invalid command \N
psql:/root/8500.sql:4185: invalid command \N
psql:/root/8500.sql:4186: invalid command \N
psql:/root/8500.sql:4233: invalid command \N
psql:/root/8500.sql:4234: invalid command \N
psql:/root/8500.sql:4235: invalid command \N
psql:/root/8500.sql:4283: invalid command \N
psql:/root/8500.sql:4284: invalid command \N
psql:/root/8500.sql:4285: invalid command \N
psql:/root/8500.sql:4334: invalid command \N
psql:/root/8500.sql:4335: invalid command \N
psql:/root/8500.sql:4336: invalid command \N
psql:/root/8500.sql:4375: invalid command \N
psql:/root/8500.sql:4376: invalid command \N
psql:/root/8500.sql:4377: invalid command \N
psql:/root/8500.sql:4424: invalid command \N
psql:/root/8500.sql:4425: invalid command \N
psql:/root/8500.sql:4426: invalid command \N
psql:/root/8500.sql:4471: invalid command \N
psql:/root/8500.sql:4472: invalid command \N
psql:/root/8500.sql:4473: invalid command \N
psql:/root/8500.sql:4521: invalid command \N
psql:/root/8500.sql:4522: invalid command \N
psql:/root/8500.sql:4523: invalid command \N
psql:/root/8500.sql:4560: invalid command \N
psql:/root/8500.sql:4561: invalid command \N
psql:/root/8500.sql:4562: invalid command \N
psql:/root/8500.sql:4605: invalid command \N
psql:/root/8500.sql:4606: invalid command \N
psql:/root/8500.sql:4607: invalid command \N
psql:/root/8500.sql:4655: invalid command \N
psql:/root/8500.sql:4656: invalid command \N
psql:/root/8500.sql:4657: invalid command \N
psql:/root/8500.sql:4702: invalid command \N
psql:/root/8500.sql:4703: invalid command \N
psql:/root/8500.sql:4704: invalid command \N
psql:/root/8500.sql:4746: invalid command \N
psql:/root/8500.sql:4747: invalid command \N
psql:/root/8500.sql:4748: invalid command \N
psql:/root/8500.sql:4790: invalid command \N
psql:/root/8500.sql:4791: invalid command \N
psql:/root/8500.sql:4792: invalid command \N
psql:/root/8500.sql:4834: invalid command \.
psql:/root/8500.sql:4935: invalid command \.
psql:/root/8500.sql:5036: invalid command \.
psql:/root/8500.sql:5137: invalid command \.
psql:/root/8500.sql:5237: invalid command \.
psql:/root/8500.sql:5240: ERROR:  syntax error at or near "8500"
LINE 1: 8500 a 040 sacat