[OPEN-ILS-GENERAL] Import issues

Fri Aug 7 11:50:03 EDT 2009

Hi Dan,

Many thanks to you for your kind response and testing our library's sample
records which I greatly appreciate. I followed your instructions with the
options '--encoding UTF8 --idfield 852 --idsubfield x' to a new set of
records, and I was quite happy to see the result that It imported all the 97
records from the file! After the successful import, I applied the same
options again to the different sets of records. But, after the first
successful SQL execution, while executing every other generated SQL, it gave
error like "psql:/root/8400.sql:92: ERROR:  duplicate key violates unique
constraint "biblio_record_unique_tcn"...".I thought the option 'idfield 852
--idsubfield x' generated the keys that would cause duplicate constraint.
So, I tried without those options as well and only with' --encoding UTF8',
but it didn't work.

And yes, I also tested the '8000.mrc' again, and it only parsed 40 records.
But, the records were not imported because of the same SQL error as above. I
tried several times but failed to execute the SQL file. I will be validating
the MARC records and will try importing again. I will let you know the
result, soon.

I executed the following commands while importing the bibliographic record
'8400.mrc' and I did the same by replacing the existing file name with the
respective file name:

#perl marc2bre.pl --db_user postgres --db_host localhost --db_pw evergreen
--db_name evergreen --encoding UTF8 --idfield 852 --idsubfield x 8400.mrc >
~/8400.bre

OR

#perl marc2bre.pl --db_user postgres --db_host localhost --db_pw evergreen
--db_name evergreen --encoding UTF8 8400.mrc > ~/8400.bre

#perl direct_ingest.pl ~/8400.bre > ~/8400.ingest
#perl pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or msfe -or
mkfe -or msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a msefe
--output=8400.sql < ~/8400.ingest
#cp 8400.sql ~
#psql -U evergreen evergreen
#\i ~/8400.sql
# \i
/home/opensrf/Evergreen-ILS-1.4.0.4/Open-ILS/src/extras/import/quick_metarecord_map.sql

I have attached the errors that were shown herewith, as the errors were
quite long.

I look forward to hearing from you.

Thank you.

--
Dibyendra Hyoju
Madan Puraskar Pustakalaya
Lalitpur, Nepal

On Fri, Aug 7, 2009 at 9:45 AM, Dan Scott <denials at gmail.com> wrote:

> Hi Dibyendra:
>
> First, I think you'll want to tell marc2bre.pl that you're working
> with records encoded in UTF8, or it will try to convert them from
> MARC8 to UTF8 (and that will really mess up your records). Just add
> the "--encoding UTF8" flag to your marc2bre.pl options to do that. You
> may also want to add "--idfield 852 --idsubfield x" if you want to
> force your records in Evergreen to have record IDs that match your
> accession numbers in the old system - just a friendly suggestion.
>
> When I ran marc2bre.pl / direct_ingest.pl against your 8000.mrc file,
> I only had 40 records out of the original 100 make it through the
> ingest stage. It looks like you're running into a limitation of the
> fingerprint algorithm in /openils/var/catalog/biblio_fingerprint.js
> (which, as a gross simplification, largely tries to concatenate author
> + title together to identify works) -- the fingerprint algorithm
> doesn't understand anything except for ASCII; all non-ASCII characters
> end up being thrown away for the purposes of the fingerprint. If the
> first character of the author or title subfields that the algorithm
> finds are not in plain ASCII, it generates no fingerprint, and the
> record can't be ingested.
>
> I don't have a quick fix for you, unfortunately. It's a significant
> problem.
>
> Dan
>
> 2009/8/6 Dibyendra Hyoju <dibyendra at gmail.com>:
> > Hello everyone,
> >
> > Today, I again tried to import a new set of parallel records, which I
> > thought there would be no errors. But, I am getting messages like 'Use of
> > uninitialized value $value in substitution (s///) at marc2bre.pl line
> 412.'
> > repeatedly while executing 'perl marc2bre.pl --db_user evergreen
> --db_host
> > localhost --db_pw evergreen --db_name evergreen 5300.mrc > ~/5300.bre'.
> >
> > I continued converting the bre into ingest file ignoring the message. But
> I
> > got error like 'Couldn't process record...'. I've attached the error
> message
> > herewith as the error message was very long, and a record which I had
> tried
> > is also attached herewith. If anyone has faced similar problems and had
> > already found the solution, please suggest. I would be very much
> thankful.
> >
> > Any help will be appreciated.
> >
> > Thank you.
> >
> > With kind regards,
> > Dibyendra
> >
> >
> > On Wed, Aug 5, 2009 at 5:06 PM, Dibyendra Hyoju <dibyendra at gmail.com>
> wrote:
> >>
> >> Hi all,
> >>
> >> I have finished installing Evergreen 1.4 on Debian-lenny, and trying to
> >> import MARC records. I have successfully imported the gutenberg records
> as
> >> shown in the Importing bibliographic records. But, I couldn't do the
> same
> >> for our library records which have both local script and roman
> >> transliteration. I am getting lots of similar warnings while converting
> the
> >> MARC records into Evegreen BRE JSON format. Like I am getting 'no
> mapping
> >> found for [0xCC] at position 3 in Bhārgav Bhūshaṇ Presa)
> g0=ASCII_DEFAULT
> >> g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 210.'  while
> >> executing 'perl marc2bre.pl --db_user evergreen --db_host localhost
> --db_pw
> >> evergreen --db_name evergreen 8500.mrc > ~/8500.bre'. Though, it
> generates
> >> Open-ILS JSON ingest file after executing perl direct_ingest.pl
> ~/8000.bre >
> >> ~/8000.ingest'.  But, it couldn't generate a SQL file after executing
> 'perl
> >> pg_loader.pl -or bre -or mrd -or mfr -or mtfe -or mafe -or msfe -or mkfe
> -or
> >> msefe -a mrd -a mfr -a mtfe -a mafe -a msfe -a mkfe -a msefe
> >> --output=~/8500.sql < ~/8500.ingest'. I followed the same process for
> the
> >> gutenberg records and they were successfully imported.
> >>
> >> I've also attached the file containing some of our library records if
> >> somebody is interested to import the records for testing. If it can be
> >> imported by any other methods, please let me know. Any help will be
> highly
> >> appreciated.
> >>
> >> Thank you.
> >>
> >> --
> >> Dibyendra Hyoju
> >> Madan Puraskar Pustakalaya
> >> Lalitpur, Nepal
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Dibyendra
> >
>
>
>
> --
> Dan Scott
> Laurentian University
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://libmail.georgialibraries.org/pipermail/open-ils-general/attachments/20090807/bf8c4f98/attachment-0001.htm 
-------------- next part --------------
SET
BEGIN
psql:/root/8400.sql:92: ERROR:  duplicate key violates unique constraint "biblio_record_unique_tcn"
CONTEXT:  COPY record_entry, line 1: "t	now	1	f	now	1	nepalilokakathatulasi	8600	IMPORT-1249655925.13782	<record    xmlns:xsi="http://www...."
psql:/root/8400.sql:94: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8400.sql:96: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8400.sql:97: invalid command \N
psql:/root/8400.sql:98: invalid command \N
psql:/root/8400.sql:99: invalid command \N
psql:/root/8400.sql:100: invalid command \N
psql:/root/8400.sql:101: invalid command \N
psql:/root/8400.sql:102: invalid command \N
psql:/root/8400.sql:103: invalid command \N
psql:/root/8400.sql:104: invalid command \N
psql:/root/8400.sql:105: invalid command \N
psql:/root/8400.sql:106: invalid command \N
psql:/root/8400.sql:107: invalid command \N
psql:/root/8400.sql:108: invalid command \N
psql:/root/8400.sql:109: invalid command \N
psql:/root/8400.sql:110: invalid command \N
psql:/root/8400.sql:111: invalid command \N
psql:/root/8400.sql:112: invalid command \N
psql:/root/8400.sql:113: invalid command \N
psql:/root/8400.sql:114: invalid command \N
psql:/root/8400.sql:115: invalid command \N
psql:/root/8400.sql:116: invalid command \N
psql:/root/8400.sql:117: invalid command \N
psql:/root/8400.sql:118: invalid command \N
psql:/root/8400.sql:119: invalid command \N
psql:/root/8400.sql:120: invalid command \N
psql:/root/8400.sql:121: invalid command \N
psql:/root/8400.sql:122: invalid command \N
psql:/root/8400.sql:123: invalid command \N
psql:/root/8400.sql:124: invalid command \N
psql:/root/8400.sql:125: invalid command \N
psql:/root/8400.sql:126: invalid command \N
psql:/root/8400.sql:127: invalid command \N
psql:/root/8400.sql:128: invalid command \N
psql:/root/8400.sql:129: invalid command \N
psql:/root/8400.sql:130: invalid command \N
psql:/root/8400.sql:131: invalid command \N
psql:/root/8400.sql:132: invalid command \N
psql:/root/8400.sql:133: invalid command \N
psql:/root/8400.sql:134: invalid command \N
psql:/root/8400.sql:135: invalid command \N
psql:/root/8400.sql:136: invalid command \N
psql:/root/8400.sql:137: invalid command \N
psql:/root/8400.sql:138: invalid command \N
psql:/root/8400.sql:139: invalid command \N
psql:/root/8400.sql:140: invalid command \N
psql:/root/8400.sql:141: invalid command \N
psql:/root/8400.sql:142: invalid command \N
psql:/root/8400.sql:143: invalid command \N
psql:/root/8400.sql:144: invalid command \N
psql:/root/8400.sql:145: invalid command \N
psql:/root/8400.sql:146: invalid command \N
psql:/root/8400.sql:147: invalid command \N
psql:/root/8400.sql:148: invalid command \N
psql:/root/8400.sql:149: invalid command \N
psql:/root/8400.sql:150: invalid command \N
psql:/root/8400.sql:151: invalid command \N
psql:/root/8400.sql:152: invalid command \N
psql:/root/8400.sql:153: invalid command \N
psql:/root/8400.sql:154: invalid command \N
psql:/root/8400.sql:155: invalid command \N
psql:/root/8400.sql:156: invalid command \N
psql:/root/8400.sql:157: invalid command \N
psql:/root/8400.sql:158: invalid command \N
psql:/root/8400.sql:159: invalid command \N
psql:/root/8400.sql:160: invalid command \N
psql:/root/8400.sql:161: invalid command \N
psql:/root/8400.sql:162: invalid command \N
psql:/root/8400.sql:163: invalid command \N
psql:/root/8400.sql:164: invalid command \N
psql:/root/8400.sql:165: invalid command \N
psql:/root/8400.sql:166: invalid command \N
psql:/root/8400.sql:167: invalid command \N
psql:/root/8400.sql:168: invalid command \N
psql:/root/8400.sql:169: invalid command \N
psql:/root/8400.sql:170: invalid command \N
psql:/root/8400.sql:171: invalid command \N
psql:/root/8400.sql:172: invalid command \N
psql:/root/8400.sql:173: invalid command \N
psql:/root/8400.sql:174: invalid command \N
psql:/root/8400.sql:175: invalid command \N
psql:/root/8400.sql:176: invalid command \N
psql:/root/8400.sql:177: invalid command \N
psql:/root/8400.sql:178: invalid command \N
psql:/root/8400.sql:179: invalid command \N
psql:/root/8400.sql:180: invalid command \N
psql:/root/8400.sql:181: invalid command \N
psql:/root/8400.sql:182: invalid command \N
psql:/root/8400.sql:183: invalid command \.
psql:/root/8400.sql:186: ERROR:  syntax error at or near "f"
LINE 1: f m a 
        ^
psql:/root/8400.sql:187: ERROR:  current transaction is aborted, commands ignored until end of transaction block
psql:/root/8400.sql:188: invalid command \N
psql:/root/8400.sql:189: invalid command \N
psql:/root/8400.sql:190: invalid command \N
psql:/root/8400.sql:238: invalid command \N
psql:/root/8400.sql:239: invalid command \N
psql:/root/8400.sql:240: invalid command \N
psql:/root/8400.sql:291: invalid command \N
psql:/root/8400.sql:292: invalid command \N
psql:/root/8400.sql:293: invalid command \N
psql:/root/8400.sql:630: invalid command \N
psql:/root/8400.sql:631: invalid command \N
psql:/root/8400.sql:632: invalid command \N
psql:/root/8400.sql:1072: invalid command \N
psql:/root/8400.sql:1073: invalid command \N
psql:/root/8400.sql:1074: invalid command \N
psql:/root/8400.sql:1120: invalid command \N
psql:/root/8400.sql:1121: invalid command \N
psql:/root/8400.sql:1122: invalid command \N
psql:/root/8400.sql:1182: invalid command \N
psql:/root/8400.sql:1183: invalid command \N
psql:/root/8400.sql:1184: invalid command \N
psql:/root/8400.sql:1229: invalid command \N
psql:/root/8400.sql:1230: invalid command \N
psql:/root/8400.sql:1231: invalid command \N
psql:/root/8400.sql:1281: invalid command \N
psql:/root/8400.sql:1282: invalid command \N
psql:/root/8400.sql:1283: invalid command \N
psql:/root/8400.sql:1332: invalid command \N
psql:/root/8400.sql:1333: invalid command \N
psql:/root/8400.sql:1334: invalid command \N
psql:/root/8400.sql:1375: invalid command \N
psql:/root/8400.sql:1376: invalid command \N
psql:/root/8400.sql:1377: invalid command \N
psql:/root/8400.sql:1424: invalid command \N
psql:/root/8400.sql:1425: invalid command \N
psql:/root/8400.sql:1426: invalid command \N
psql:/root/8400.sql:1472: invalid command \N
psql:/root/8400.sql:1473: invalid command \N
psql:/root/8400.sql:1474: invalid command \N
psql:/root/8400.sql:1527: invalid command \N
psql:/root/8400.sql:1528: invalid command \N
psql:/root/8400.sql:1529: invalid command \N
psql:/root/8400.sql:1576: invalid command \N
psql:/root/8400.sql:1577: invalid command \N
psql:/root/8400.sql:1578: invalid command \N
psql:/root/8400.sql:1624: invalid command \N
psql:/root/8400.sql:1625: invalid command \N
psql:/root/8400.sql:1626: invalid command \N
psql:/root/8400.sql:1664: invalid command \N
psql:/root/8400.sql:1665: invalid command \N
psql:/root/8400.sql:1666: invalid command \N
psql:/root/8400.sql:1711: invalid command \N
psql:/root/8400.sql:1712: invalid command \N
psql:/root/8400.sql:1713: invalid command \N
psql:/root/8400.sql:1754: invalid command \N
psql:/root/8400.sql:1755: invalid command \N
psql:/root/8400.sql:1756: invalid command \N
psql:/root/8400.sql:1795: invalid command \N
psql:/root/8400.sql:1796: invalid command \N
psql:/root/8400.sql:1797: invalid command \N
psql:/root/8400.sql:1842: invalid command \N
psql:/root/8400.sql:1843: invalid command \N
psql:/root/8400.sql:1844: invalid command \N
psql:/root/8400.sql:1894: invalid command \N
psql:/root/8400.sql:1895: invalid command \N
psql:/root/8400.sql:1896: invalid command \N
psql:/root/8400.sql:1944: invalid command \N
psql:/root/8400.sql:1945: invalid command \N
psql:/root/8400.sql:1946: invalid command \N
psql:/root/8400.sql:1996: invalid command \N
psql:/root/8400.sql:1997: invalid command \N
psql:/root/8400.sql:1998: invalid command \N
psql:/root/8400.sql:2043: invalid command \N
psql:/root/8400.sql:2044: invalid command \N
psql:/root/8400.sql:2045: invalid command \N
psql:/root/8400.sql:2095: invalid command \N
psql:/root/8400.sql:2096: invalid command \N
psql:/root/8400.sql:2097: invalid command \N
psql:/root/8400.sql:2143: invalid command \N
psql:/root/8400.sql:2144: invalid command \N
psql:/root/8400.sql:2145: invalid command \N
psql:/root/8400.sql:2194: invalid command \N
psql:/root/8400.sql:2195: invalid command \N
psql:/root/8400.sql:2196: invalid command \N
psql:/root/8400.sql:2244: invalid command \N
psql:/root/8400.sql:2245: invalid command \N
psql:/root/8400.sql:2246: invalid command \N
psql:/root/8400.sql:2291: invalid command \N
psql:/root/8400.sql:2292: invalid command \N
psql:/root/8400.sql:2293: invalid command \N
psql:/root/8400.sql:2345: invalid command \N
psql:/root/8400.sql:2346: invalid command \N
psql:/root/8400.sql:2347: invalid command \N
psql:/root/8400.sql:2395: invalid command \N
psql:/root/8400.sql:2396: invalid command \N
psql:/root/8400.sql:2397: invalid command \N
psql:/root/8400.sql:2441: invalid command \N
psql:/root/8400.sql:2442: invalid command \N
psql:/root/8400.sql:2443: invalid command \N
psql:/root/8400.sql:2487: invalid command \N
psql:/root/8400.sql:2488: invalid command \N
psql:/root/8400.sql:2489: invalid command \N
psql:/root/8400.sql:2545: invalid command \N
psql:/root/8400.sql:2546: invalid command \N
psql:/root/8400.sql:2547: invalid command \N
psql:/root/8400.sql:2596: invalid command \N
psql:/root/8400.sql:2597: invalid command \N
psql:/root/8400.sql:2598: invalid command \N
psql:/root/8400.sql:2649: invalid command \N
psql:/root/8400.sql:2650: invalid command \N
psql:/root/8400.sql:2651: invalid command \N
psql:/root/8400.sql:2690: invalid command \N
psql:/root/8400.sql:2691: invalid command \N
psql:/root/8400.sql:2692: invalid command \N
psql:/root/8400.sql:2738: invalid command \N
psql:/root/8400.sql:2739: invalid command \N
psql:/root/8400.sql:2740: invalid command \N
psql:/root/8400.sql:2790: invalid command \N
psql:/root/8400.sql:2791: invalid command \N
psql:/root/8400.sql:2792: invalid command \N
psql:/root/8400.sql:2841: invalid command \N
psql:/root/8400.sql:2842: invalid command \N
psql:/root/8400.sql:2843: invalid command \N
psql:/root/8400.sql:2893: invalid command \N
psql:/root/8400.sql:2894: invalid command \N
psql:/root/8400.sql:2895: invalid command \N
psql:/root/8400.sql:2942: invalid command \N
psql:/root/8400.sql:2943: invalid command \N
psql:/root/8400.sql:2944: invalid command \N
psql:/root/8400.sql:2972: invalid command \N
psql:/root/8400.sql:2973: invalid command \N
psql:/root/8400.sql:2974: invalid command \N
psql:/root/8400.sql:3021: invalid command \N
psql:/root/8400.sql:3022: invalid command \N
psql:/root/8400.sql:3023: invalid command \N
psql:/root/8400.sql:3069: invalid command \N
psql:/root/8400.sql:3070: invalid command \N
psql:/root/8400.sql:3071: invalid command \N
psql:/root/8400.sql:3120: invalid command \N
psql:/root/8400.sql:3121: invalid command \N
psql:/root/8400.sql:3122: invalid command \N
psql:/root/8400.sql:3171: invalid command \N
psql:/root/8400.sql:3172: invalid command \N
psql:/root/8400.sql:3173: invalid command \N
psql:/root/8400.sql:3223: invalid command \N
psql:/root/8400.sql:3224: invalid command \N
psql:/root/8400.sql:3225: invalid command \N
psql:/root/8400.sql:3269: invalid command \N
psql:/root/8400.sql:3270: invalid command \N
psql:/root/8400.sql:3271: invalid command \N
psql:/root/8400.sql:3320: invalid command \N
psql:/root/8400.sql:3321: invalid command \N
psql:/root/8400.sql:3322: invalid command \N
psql:/root/8400.sql:3366: invalid command \N
psql:/root/8400.sql:3367: invalid command \N
psql:/root/8400.sql:3368: invalid command \N
psql:/root/8400.sql:3412: invalid command \N
psql:/root/8400.sql:3413: invalid command \N
psql:/root/8400.sql:3414: invalid command \N
psql:/root/8400.sql:3462: invalid command \N
psql:/root/8400.sql:3463: invalid command \N
psql:/root/8400.sql:3464: invalid command \N
psql:/root/8400.sql:3512: invalid command \N
psql:/root/8400.sql:3513: invalid command \N
psql:/root/8400.sql:3514: invalid command \N
psql:/root/8400.sql:3566: invalid command \N
psql:/root/8400.sql:3567: invalid command \N
psql:/root/8400.sql:3568: invalid command \N
psql:/root/8400.sql:3619: invalid command \N
psql:/root/8400.sql:3620: invalid command \N
psql:/root/8400.sql:3621: invalid command \N
psql:/root/8400.sql:3670: invalid command \N
psql:/root/8400.sql:3671: invalid command \N
psql:/root/8400.sql:3672: invalid command \N
psql:/root/8400.sql:3724: invalid command \N
psql:/root/8400.sql:3725: invalid command \N
psql:/root/8400.sql:3726: invalid command \N
psql:/root/8400.sql:3775: invalid command \N
psql:/root/8400.sql:3776: invalid command \N
psql:/root/8400.sql:3777: invalid command \N
psql:/root/8400.sql:3830: invalid command \N
psql:/root/8400.sql:3831: invalid command \N
psql:/root/8400.sql:3832: invalid command \N
psql:/root/8400.sql:3875: invalid command \N
psql:/root/8400.sql:3876: invalid command \N
psql:/root/8400.sql:3877: invalid command \N
psql:/root/8400.sql:3927: invalid command \N
psql:/root/8400.sql:3928: invalid command \N
psql:/root/8400.sql:3929: invalid command \N
psql:/root/8400.sql:3992: invalid command \N
psql:/root/8400.sql:3993: invalid command \N
psql:/root/8400.sql:3994: invalid command \N
psql:/root/8400.sql:4043: invalid command \N
psql:/root/8400.sql:4044: invalid command \N
psql:/root/8400.sql:4045: invalid command \N
psql:/root/8400.sql:4088: invalid command \N
psql:/root/8400.sql:4089: invalid command \N
psql:/root/8400.sql:4090: invalid command \N
psql:/root/8400.sql:4142: invalid command \N
psql:/root/8400.sql:4143: invalid command \N
psql:/root/8400.sql:4144: invalid command \N
psql:/root/8400.sql:4194: invalid command \N
psql:/root/8400.sql:4195: invalid command \N
psql:/root/8400.sql:4196: invalid command \N
psql:/root/8400.sql:4240: invalid command \N
psql:/root/8400.sql:4241: invalid command \N
psql:/root/8400.sql:4242: invalid command \N
psql:/root/8400.sql:4290: invalid command \N
psql:/root/8400.sql:4291: invalid command \N
psql:/root/8400.sql:4292: invalid command \N
psql:/root/8400.sql:4341: invalid command \N
psql:/root/8400.sql:4342: invalid command \N
psql:/root/8400.sql:4343: invalid command \N
psql:/root/8400.sql:4395: invalid command \N
psql:/root/8400.sql:4396: invalid command \N
psql:/root/8400.sql:4397: invalid command \N
psql:/root/8400.sql:4445: invalid command \.
psql:/root/8400.sql:4535: invalid command \.
psql:/root/8400.sql:4620: invalid command \.
psql:/root/8400.sql:4707: invalid command \.
psql:/root/8400.sql:4799: ERROR:  syntax error at or near "8600"
LINE 1: 8600 a 040 sacat