[OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.

[Dale Arntson]

Mike,

On first inspection, it looks like rerunning the import chain and 
filtering on mrd records will be the most straight forward for us. 
Thanks for considering all of our options. I'll report back about how 
long it takes to do a repair like this.

-dale

Mike Rylander wrote:
> On Fri, Jun 12, 2009 at 10:36 AM, Frances Dean
> McNamara<fdmcnama at uchicago.edu> wrote:
>> I believe NONE of the records loaded got rows.  Perhaps trying the bib_ids.list for ranges of records to see if it works.  You say that would be a really long thing?  Does that do more thand the metabib.rec_descriptor inserts?
>>
> 
> Right.  I gave you the more complicated SQL so you could grab a fresh
> set of just the remaining to-be-fixed records if you have to stop the
> process or it dies.  It re-ingests the entire record, basically
> rewriting it.  This, of course, assumes that the mrd portion of the
> ingest process is not broken in some way and that re-ingesting will
> fix things. you can test on one record to find out, of course.
> 
>> Or we could use your first suggestion.  We actually do have the original input files broken down to about 59 files I think.  I will talk to Dale when he is in.  If we figure out which way to do it we can probably script it.
>>
> 
> This will be faster, but more involved.  Pop in here if you run into trouble.
> 
> --miker
> 
>> Thanks.
>>
>> Frances
>>
>> -----Original Message-----
>> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Mike Rylander
>> Sent: Friday, June 12, 2009 9:03 AM
>> To: Evergreen Development Discussion List
>> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>>
>> On Fri, Jun 12, 2009 at 9:47 AM, Frances Dean
>> McNamara<fdmcnama at uchicago.edu> wrote:
>>> Ah, I see, it is pulling out stuff like type of date, date1, date2 from the fixed field, bib lvl, enc lvl, looks like mainly 008 type of data?  Is there a script we could run for a while to build this?  We have over 5 million bibs.  Thanks.  Frances
>>>
>> You can use the stock ingest scripts, and avoid intermediate files by
>> doing something like:
>>
>> $ marc2bre.pl --marctype=xml [... other options] | direct_ingest.pl |
>> grep '"__c":"mrd"' > mrd.ingest
>>
>> for 1.4.  If you're working with 1.2, you'll need to change the grep,
>> but that will get you the mrd data.
>>
>> Another other option is to get a list of all the bibs that lack 'mrd's
>> in a file and re-ingest them.  In psql:
>>
>>  \t
>>  \o /tmp/bib_ids.list
>>  SELECT b.id FROM biblio.record_entry b LEFT JOIN
>> metabib.rec_descriptor d ON (b.id = d.record) WHERE r.id IS NULL b.id
>>> 0 AND NOT b.deleted;
>>  \o
>>  \t
>>
>> Then as the opensrf user, at the command line:
>>
>> $ for i in `cat /tmp/bib_ids.list`; do echo "request open-ils.ingest
>> open-ils.ingest.full.biblio.record $i"; done | srfsh
>>
>> That will take, um, a really long time to complete.  If anything
>> happens to stop that, just start over with a fresh bib_ids.list file
>> -- the SQL above will give you only te records that are missing an
>> 'mrd'.
>>
>> --miker
>>
>>> -----Original Message-----
>>> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Frances Dean McNamara
>>> Sent: Friday, June 12, 2009 8:37 AM
>>> To: Evergreen Development Discussion List
>>> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>>>
>>> Dale has not replied to this yet, but I don't think we have kept the interim files due to file space (big database).
>>>
>>> Are their scripts needed to regenerate that table from what is in the db now?  What is in that table anyhow?
>>>
>>> Frances McNamara
>>> University of Chicago
>>>
>>> -----Original Message-----
>>> From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Mike Rylander
>>> Sent: Friday, June 12, 2009 7:12 AM
>>> To: Evergreen Development Discussion List
>>> Subject: Re: [OPEN-ILS-DEV] No rows in metabib.rec_descriptor after loading bibs.
>>>
>>> On Thu, Jun 11, 2009 at 7:23 PM, Dale Arntson<arnt at uchicago.edu> wrote:
>>>> Hi All,
>>>>
>>>> After loading our bib records into evergreen, I got zero hits on searches in
>>>> the evergreen client. I traced the problem back to the fact that there are
>>>> no records in the metabib.rec_descriptor table. The other metabib tables
>>>> seem fully populated. Here are the flags I used in parallel_pg_loader.
>>>>
>>>> perl parallel_pg_loader.pl -order bre -order mrd -order mfr -order mtfe
>>>>  -order mafe -order msfe -order mkfe -order msefe -autoprimary mrd
>>>> -autoprimary mfr -autoprimary mtfe -autoprimary mafe -autoprimary msfe
>>>> -autoprimary mkfe -autoprimary msefe
>>>>
>>>> Any ideas what I did wrong? Any ideas how to fix it? The bibs took a long
>>>> time to load. I would rather not redo it, if I don't have to.
>>> If you have the intermediate bib processing files, particularly the
>>> output of direct_ingest, make sure that file contains 'mrd' rows.  For
>>> OpenSRF 0.9 (Evergreen 1.2.x), those will start with '/*--S mrd--*/',
>>> and in OpenSRF 1.0.x they will start with '[{"__c":"mrd","__p"'.
>>>
>>> If you can find that file and it contains 'mrd' lines, then we can
>>> simply regenerate that table's worth of data. Otherwise you'll need to
>>> reprocess the bib records in order to get that data, but you won't
>>> need to reload the entire dataset.
>>>
>>> --
>>> Mike Rylander
>>>  | VP, Research and Design
>>>  | Equinox Software, Inc. / The Evergreen Experts
>>>  | phone:  1-877-OPEN-ILS (673-6457)
>>>  | email:  miker at esilibrary.com
>>>  | web:  http://www.esilibrary.com
>>>
>>
>>
>> --
>> Mike Rylander
>>  | VP, Research and Design
>>  | Equinox Software, Inc. / The Evergreen Experts
>>  | phone:  1-877-OPEN-ILS (673-6457)
>>  | email:  miker at esilibrary.com
>>  | web:  http://www.esilibrary.com
>>
> 
> 
>