[Evergreen-catalogers] Vandelay and large files

[Kathy Lussier]

Hi Mary,

I believe the MassLNC consortium that is live on Evergreen is typically 
loading files with fewer than 100 records. From what I understand, they 
are not having problems with load times.

We do have another consortium, though, that is experiencing problems 
with load times for any size file if they are using a match set that 
matches on the 020, 022 or 024 field. I don't know if it's related, but, 
during the development, we had asked for some normalization on these 
fields (the 020 field in particular) so that incoming records could 
match when there is pbk or other trailing text at the end of the field.

Since this problem is happening in one Evergreen instance but not the 
other, we're looking at the possibility that we may be missing an index. 
I don't know if there are any other system configuration issues we 
should be looking at. As soon as we identify the source of the problem, 
I'll be happy to share it with the list.

FWIW, when I was testing, there were a couple of instances where I 
accidentally loaded files with more than 1,000 records, and, even though 
it took some time, the processing ultimately completed. However, I don't 
recall now what kind of matching I was doing.

I'm sorry I couldn't be more help on this one!

Kathy

--
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 756-0172
(508) 755-3721 (fax)
klussier at masslnc.org
Twitter: http://www.twitter.com/kmlussier

On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
> Hi Kathy,
>
> I’m cross-posting this question to the Evergreen Catalogers list.
>
> I love using the new Vandelay in our Evergreen 2.2-ish environment. But
> I’m having problems when a file is large, more than 200 bibs. The load
> screen seems to freeze at “Processing… 100”. It never gets out of the
> processing stage, and the bibs don’t get loaded. I have several large
> files (1000+) of OneClickDigital bib records that I need to load through
> the client instead of through the database, since I need the overlay
> feature to add URLs for each subscribing library. Are you experiencing
> any difficulty when loading a large file?
>
> It seems like my only option is to divide the files into smaller chunks,
> but that makes loading pretty labor intensive. I’m hoping there’s a fix
> for our loading logjam.
>
> Thanks,
>
> Mary
>