[OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.

Fri Jun 12 10:42:30 EDT 2009

On Fri, Jun 12, 2009 at 10:36 AM, Frances Dean
McNamara<fdmcnama at uchicago.edu> wrote:
> I believe NONE of the records loaded got rows.  Perhaps trying the bib_ids.list for ranges of records to see if it works.  You say that would be a really long thing?  Does that do more thand the metabib.rec_descriptor inserts?
>

Right.  I gave you the more complicated SQL so you could grab a fresh
set of just the remaining to-be-fixed records if you have to stop the
process or it dies.  It re-ingests the entire record, basically
rewriting it.  This, of course, assumes that the mrd portion of the
ingest process is not broken in some way and that re-ingesting will
fix things. you can test on one record to find out, of course.

> Or we could use your first suggestion.  We actually do have the original input files broken down to about 59 files I think.  I will talk to Dale when he is in.  If we figure out which way to do it we can probably script it.
>

This will be faster, but more involved.  Pop in here if you run into trouble.

--miker

> Thanks.
>
> Frances
>
> -----Original Message-----
> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Mike Rylander
> Sent: Friday, June 12, 2009 9:03 AM
> To: Evergreen Development Discussion List
> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>
> On Fri, Jun 12, 2009 at 9:47 AM, Frances Dean
> McNamara<fdmcnama at uchicago.edu> wrote:
>> Ah, I see, it is pulling out stuff like type of date, date1, date2 from the fixed field, bib lvl, enc lvl, looks like mainly 008 type of data?  Is there a script we could run for a while to build this?  We have over 5 million bibs.  Thanks.  Frances
>>
>
> You can use the stock ingest scripts, and avoid intermediate files by
> doing something like:
>
> $ marc2bre.pl --marctype=xml [... other options] | direct_ingest.pl |
> grep '"__c":"mrd"' > mrd.ingest
>
> for 1.4.  If you're working with 1.2, you'll need to change the grep,
> but that will get you the mrd data.
>
> Another other option is to get a list of all the bibs that lack 'mrd's
> in a file and re-ingest them.  In psql:
>
>  \t
>  \o /tmp/bib_ids.list
>  SELECT b.id FROM biblio.record_entry b LEFT JOIN
> metabib.rec_descriptor d ON (b.id = d.record) WHERE r.id IS NULL b.id
>> 0 AND NOT b.deleted;
>  \o
>  \t
>
> Then as the opensrf user, at the command line:
>
> $ for i in `cat /tmp/bib_ids.list`; do echo "request open-ils.ingest
> open-ils.ingest.full.biblio.record $i"; done | srfsh
>
> That will take, um, a really long time to complete.  If anything
> happens to stop that, just start over with a fresh bib_ids.list file
> -- the SQL above will give you only te records that are missing an
> 'mrd'.
>
> --miker
>
>> -----Original Message-----
>> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Frances Dean McNamara
>> Sent: Friday, June 12, 2009 8:37 AM
>> To: Evergreen Development Discussion List
>> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>>
>> Dale has not replied to this yet, but I don't think we have kept the interim files due to file space (big database).
>>
>> Are their scripts needed to regenerate that table from what is in the db now?  What is in that table anyhow?
>>
>> Frances McNamara
>> University of Chicago
>>
>> -----Original Message-----
>> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Mike Rylander
>> Sent: Friday, June 12, 2009 7:12 AM
>> To: Evergreen Development Discussion List
>> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>>
>> On Thu, Jun 11, 2009 at 7:23 PM, Dale Arntson<arnt at uchicago.edu> wrote:
>>> Hi All,
>>>
>>> After loading our bib records into evergreen, I got zero hits on searches in
>>> the evergreen client. I traced the problem back to the fact that there are
>>> no records in the metabib.rec_descriptor table. The other metabib tables
>>> seem fully populated. Here are the flags I used in parallel_pg_loader.
>>>
>>> perl parallel_pg_loader.pl -order bre -order mrd -order mfr -order mtfe
>>>  -order mafe -order msfe -order mkfe -order msefe -autoprimary mrd
>>> -autoprimary mfr -autoprimary mtfe -autoprimary mafe -autoprimary msfe
>>> -autoprimary mkfe -autoprimary msefe
>>>
>>> Any ideas what I did wrong? Any ideas how to fix it? The bibs took a long
>>> time to load. I would rather not redo it, if I don't have to.
>>
>> If you have the intermediate bib processing files, particularly the
>> output of direct_ingest, make sure that file contains 'mrd' rows.  For
>> OpenSRF 0.9 (Evergreen 1.2.x), those will start with '/*--S mrd--*/',
>> and in OpenSRF 1.0.x they will start with '[{"__c":"mrd","__p"'.
>>
>> If you can find that file and it contains 'mrd' lines, then we can
>> simply regenerate that table's worth of data. Otherwise you'll need to
>> reprocess the bib records in order to get that data, but you won't
>> need to reload the entire dataset.
>>
>> --
>> Mike Rylander
>>  | VP, Research and Design
>>  | Equinox Software, Inc. / The Evergreen Experts
>>  | phone:  1-877-OPEN-ILS (673-6457)
>>  | email:  miker at esilibrary.com
>>  | web:  http://www.esilibrary.com
>>
>
>
>
> --
> Mike Rylander
>  | VP, Research and Design
>  | Equinox Software, Inc. / The Evergreen Experts
>  | phone:  1-877-OPEN-ILS (673-6457)
>  | email:  miker at esilibrary.com
>  | web:  http://www.esilibrary.com
>

-- 
Mike Rylander
 | VP, Research and Design
 | Equinox Software, Inc. / The Evergreen Experts
 | phone:  1-877-OPEN-ILS (673-6457)
 | email:  miker at esilibrary.com
 | web:  http://www.esilibrary.com