[Evergreen-catalogers] Vandelay and large files

[Elizabeth McLean]

Hi Mary,

I don't know if our situation will be relevant because we are on release 2.1 and I only import records that match on the Record ID, but when Vandelay freezes and none of the records imported, there is almost always something in one of the records that Evergreen did not like. I also split files that are larger than 1,000 into smaller chunks (200-500) using Terry Reese's (free) MarcEdit program.

If records don't import, I use MarcEdit to validate the records and correct the error. If MarcEdit can't identify the problem record, I split the larger file into smaller files (100 or less). The problem record eventually causes one of the smaller files to fail. If I still can't identify the error, I split the problem file again and import. Eventually the error is revealed. I've gotten into a habit of validating and checking for recurring errors in MarcEdit before I import files in Vandelay.

As a future enhancement, I would like Vandelay to identify the problems it encounters and the records where they occur.

I hope this helps,

Libby McLean
Cataloging Librarian II
King County Library System
lmclean at kcls.org

-----Original Message-----
From: evergreen-catalogers-bounces at list.evergreen-ils.org [mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf Of Mary Llewellyn
Sent: Thursday, May 10, 2012 8:05 AM
To: 'Evergreen Community Catalogers'; 'Kathy Lussier'
Subject: Re: [Evergreen-catalogers] Vandelay and large files

Well, it was  a good news/bad news result. The good news was that with the leaner, meaner record match set (based solely on tag 001), the loader processed bibs beyond 100, and did it rapidly. The bad news is the file had
1972 bibs, and the processing screen froze at 1000. All the bibs made it to the queue, but none loaded. 

Mary


-----Original Message-----
From: evergreen-catalogers-bounces at list.evergreen-ils.org
[mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf Of Mary Llewellyn
Sent: Wednesday, May 09, 2012 4:10 PM
To: 'Kathy Lussier'
Cc: 'Evergreen Community Catalogers'
Subject: Re: [Evergreen-catalogers] Vandelay and large files

Thanks, Kathy. I'll set up another matchpoint just for these OneClickDigital (or, as I like to call them, OCD) records, since they should have the same 001, so I don't have to worry about matching on 020.

Wish me luck!

Mary


-----Original Message-----
From: Kathy Lussier [mailto:klussier at masslnc.org]
Sent: Wednesday, May 09, 2012 3:36 PM
To: Mary Llewellyn
Cc: Evergreen Community Catalogers
Subject: Re: Vandelay and large files

Hi Mary,

I believe the MassLNC consortium that is live on Evergreen is typically loading files with fewer than 100 records. From what I understand, they are not having problems with load times.

We do have another consortium, though, that is experiencing problems with load times for any size file if they are using a match set that matches on the 020, 022 or 024 field. I don't know if it's related, but, during the development, we had asked for some normalization on these fields (the 020 field in particular) so that incoming records could match when there is pbk or other trailing text at the end of the field.

Since this problem is happening in one Evergreen instance but not the other, we're looking at the possibility that we may be missing an index. 
I don't know if there are any other system configuration issues we should be looking at. As soon as we identify the source of the problem, I'll be happy to share it with the list.

FWIW, when I was testing, there were a couple of instances where I accidentally loaded files with more than 1,000 records, and, even though it took some time, the processing ultimately completed. However, I don't recall now what kind of matching I was doing.

I'm sorry I couldn't be more help on this one!

Kathy

--
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 756-0172
(508) 755-3721 (fax)
klussier at masslnc.org
Twitter: http://www.twitter.com/kmlussier

On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
> Hi Kathy,
>
> I'm cross-posting this question to the Evergreen Catalogers list.
>
> I love using the new Vandelay in our Evergreen 2.2-ish environment. 
> But I'm having problems when a file is large, more than 200 bibs. The 
> load screen seems to freeze at "Processing. 100". It never gets out of 
> the processing stage, and the bibs don't get loaded. I have several 
> large files (1000+) of OneClickDigital bib records that I need to load 
> through the client instead of through the database, since I need the 
> overlay feature to add URLs for each subscribing library. Are you 
> experiencing any difficulty when loading a large file?
>
> It seems like my only option is to divide the files into smaller 
> chunks, but that makes loading pretty labor intensive. I'm hoping 
> there's a fix for our loading logjam.
>
> Thanks,
>
> Mary
>

_______________________________________________
Evergreen-catalogers mailing list
Evergreen-catalogers at list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers

_______________________________________________
Evergreen-catalogers mailing list
Evergreen-catalogers at list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers