[Evergreen-catalogers] Vandelay and large files

[Mary Llewellyn]

Well, it was  a good news/bad news result. The good news was that with the
leaner, meaner record match set (based solely on tag 001), the loader
processed bibs beyond 100, and did it rapidly. The bad news is the file had
1972 bibs, and the processing screen froze at 1000. All the bibs made it to
the queue, but none loaded. 

Mary


-----Original Message-----
From: evergreen-catalogers-bounces at list.evergreen-ils.org
[mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf Of
Mary Llewellyn
Sent: Wednesday, May 09, 2012 4:10 PM
To: 'Kathy Lussier'
Cc: 'Evergreen Community Catalogers'
Subject: Re: [Evergreen-catalogers] Vandelay and large files

Thanks, Kathy. I'll set up another matchpoint just for these OneClickDigital
(or, as I like to call them, OCD) records, since they should have the same
001, so I don't have to worry about matching on 020.

Wish me luck!

Mary


-----Original Message-----
From: Kathy Lussier [mailto:klussier at masslnc.org]
Sent: Wednesday, May 09, 2012 3:36 PM
To: Mary Llewellyn
Cc: Evergreen Community Catalogers
Subject: Re: Vandelay and large files

Hi Mary,

I believe the MassLNC consortium that is live on Evergreen is typically
loading files with fewer than 100 records. From what I understand, they are
not having problems with load times.

We do have another consortium, though, that is experiencing problems with
load times for any size file if they are using a match set that matches on
the 020, 022 or 024 field. I don't know if it's related, but, during the
development, we had asked for some normalization on these fields (the 020
field in particular) so that incoming records could match when there is pbk
or other trailing text at the end of the field.

Since this problem is happening in one Evergreen instance but not the other,
we're looking at the possibility that we may be missing an index. 
I don't know if there are any other system configuration issues we should be
looking at. As soon as we identify the source of the problem, I'll be happy
to share it with the list.

FWIW, when I was testing, there were a couple of instances where I
accidentally loaded files with more than 1,000 records, and, even though it
took some time, the processing ultimately completed. However, I don't recall
now what kind of matching I was doing.

I'm sorry I couldn't be more help on this one!

Kathy

--
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 756-0172
(508) 755-3721 (fax)
klussier at masslnc.org
Twitter: http://www.twitter.com/kmlussier

On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
> Hi Kathy,
>
> I'm cross-posting this question to the Evergreen Catalogers list.
>
> I love using the new Vandelay in our Evergreen 2.2-ish environment. 
> But I'm having problems when a file is large, more than 200 bibs. The 
> load screen seems to freeze at "Processing. 100". It never gets out of 
> the processing stage, and the bibs don't get loaded. I have several 
> large files (1000+) of OneClickDigital bib records that I need to load 
> through the client instead of through the database, since I need the 
> overlay feature to add URLs for each subscribing library. Are you 
> experiencing any difficulty when loading a large file?
>
> It seems like my only option is to divide the files into smaller 
> chunks, but that makes loading pretty labor intensive. I'm hoping 
> there's a fix for our loading logjam.
>
> Thanks,
>
> Mary
>

_______________________________________________
Evergreen-catalogers mailing list
Evergreen-catalogers at list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers