[Evergreen-catalogers] Vandelay and large files

[Laurie]

Hi Mary and Kathy,

At MVLC we have been doing uploads separately as well as doing the 
process together. When doing them separately the import/export process 
does seem to do larger queues more quickly, while together it does bog 
down. I think that is what happens with the CWMars consortium because 
they do much larger queues than we do. I have done separate queues of 
under 200 records that did process quickly, so that might be helpful.

What this means is that we do the upload first and then the import 
choosing whichever Record Import Action is most practical for your needs.

Hope this helps,
Laurie



On 5/10/2012 12:00 PM, 
evergreen-catalogers-request at list.evergreen-ils.org wrote:
> Send Evergreen-catalogers mailing list submissions to
> 	evergreen-catalogers at list.evergreen-ils.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers
>
> or, via email, send a message with subject or body 'help' to
> 	evergreen-catalogers-request at list.evergreen-ils.org
>
> You can reach the person managing the list at
> 	evergreen-catalogers-owner at list.evergreen-ils.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Evergreen-catalogers digest..."
>
>
> Today's Topics:
>
>     1. Vandelay and large files (Mary Llewellyn)
>     2. Re: Vandelay and large files (Kathy Lussier)
>     3. Re: Vandelay and large files (Mary Llewellyn)
>     4. Re: Vandelay and large files (Mary Llewellyn)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 9 May 2012 14:21:46 -0400
> From: "Mary Llewellyn"<mllewell at biblio.org>
> To: "'Kathy Lussier'"<klussier at masslnc.org>, "Evergreen Community
> 	Catalogers"	<evergreen-catalogers at list.evergreen-ils.org>
> Subject: [Evergreen-catalogers] Vandelay and large files
> Message-ID:<00d501cd2e10$97b41b90$c71c52b0$@biblio.org>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Kathy,
>
>
>
> I'm cross-posting this question to the Evergreen Catalogers list.
>
>
>
> I love using the new Vandelay in our Evergreen 2.2-ish environment. But I'm
> having problems when a file is large, more than 200 bibs. The load screen
> seems to freeze at "Processing. 100". It never gets out of the processing
> stage, and the bibs don't get loaded. I have several large files (1000+)  of
> OneClickDigital bib records that I need to load through the client instead
> of through the database, since I need the overlay feature to add URLs for
> each subscribing library. Are you experiencing any difficulty when loading a
> large file?
>
>
>
> It seems like my only option is to divide the files into smaller chunks, but
> that makes loading pretty labor intensive. I'm hoping there's a fix for our
> loading logjam.
>
>
>
> Thanks,
>
>
>
> Mary
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:<http://list.evergreen-ils.org/pipermail/evergreen-catalogers/attachments/20120509/f35f5064/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 09 May 2012 15:36:15 -0400
> From: Kathy Lussier<klussier at masslnc.org>
> To: Mary Llewellyn<mllewell at biblio.org>
> Cc: Evergreen Community Catalogers
> 	<evergreen-catalogers at list.evergreen-ils.org>
> Subject: Re: [Evergreen-catalogers] Vandelay and large files
> Message-ID:<4FAAC72F.6090609 at masslnc.org>
> Content-Type: text/plain; charset=windows-1252; format=flowed
>
> Hi Mary,
>
> I believe the MassLNC consortium that is live on Evergreen is typically
> loading files with fewer than 100 records. From what I understand, they
> are not having problems with load times.
>
> We do have another consortium, though, that is experiencing problems
> with load times for any size file if they are using a match set that
> matches on the 020, 022 or 024 field. I don't know if it's related, but,
> during the development, we had asked for some normalization on these
> fields (the 020 field in particular) so that incoming records could
> match when there is pbk or other trailing text at the end of the field.
>
> Since this problem is happening in one Evergreen instance but not the
> other, we're looking at the possibility that we may be missing an index.
> I don't know if there are any other system configuration issues we
> should be looking at. As soon as we identify the source of the problem,
> I'll be happy to share it with the list.
>
> FWIW, when I was testing, there were a couple of instances where I
> accidentally loaded files with more than 1,000 records, and, even though
> it took some time, the processing ultimately completed. However, I don't
> recall now what kind of matching I was doing.
>
> I'm sorry I couldn't be more help on this one!
>
> Kathy
>
> --
> Kathy Lussier
> Project Coordinator
> Massachusetts Library Network Cooperative
> (508) 756-0172
> (508) 755-3721 (fax)
> klussier at masslnc.org
> Twitter: http://www.twitter.com/kmlussier
>
> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>> Hi Kathy,
>>
>> I?m cross-posting this question to the Evergreen Catalogers list.
>>
>> I love using the new Vandelay in our Evergreen 2.2-ish environment. But
>> I?m having problems when a file is large, more than 200 bibs. The load
>> screen seems to freeze at ?Processing? 100?. It never gets out of the
>> processing stage, and the bibs don?t get loaded. I have several large
>> files (1000+) of OneClickDigital bib records that I need to load through
>> the client instead of through the database, since I need the overlay
>> feature to add URLs for each subscribing library. Are you experiencing
>> any difficulty when loading a large file?
>>
>> It seems like my only option is to divide the files into smaller chunks,
>> but that makes loading pretty labor intensive. I?m hoping there?s a fix
>> for our loading logjam.
>>
>> Thanks,
>>
>> Mary
>>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 9 May 2012 16:10:22 -0400
> From: "Mary Llewellyn"<mllewell at biblio.org>
> To: "'Kathy Lussier'"<klussier at masslnc.org>
> Cc: 'Evergreen Community Catalogers'
> 	<evergreen-catalogers at list.evergreen-ils.org>
> Subject: Re: [Evergreen-catalogers] Vandelay and large files
> Message-ID:<00f001cd2e1f$c3bd4f40$4b37edc0$@biblio.org>
> Content-Type: text/plain;	charset="us-ascii"
>
> Thanks, Kathy. I'll set up another matchpoint just for these OneClickDigital
> (or, as I like to call them, OCD) records, since they should have the same
> 001, so I don't have to worry about matching on 020.
>
> Wish me luck!
>
> Mary
>
>
> -----Original Message-----
> From: Kathy Lussier [mailto:klussier at masslnc.org]
> Sent: Wednesday, May 09, 2012 3:36 PM
> To: Mary Llewellyn
> Cc: Evergreen Community Catalogers
> Subject: Re: Vandelay and large files
>
> Hi Mary,
>
> I believe the MassLNC consortium that is live on Evergreen is typically
> loading files with fewer than 100 records. From what I understand, they are
> not having problems with load times.
>
> We do have another consortium, though, that is experiencing problems with
> load times for any size file if they are using a match set that matches on
> the 020, 022 or 024 field. I don't know if it's related, but, during the
> development, we had asked for some normalization on these fields (the 020
> field in particular) so that incoming records could match when there is pbk
> or other trailing text at the end of the field.
>
> Since this problem is happening in one Evergreen instance but not the other,
> we're looking at the possibility that we may be missing an index.
> I don't know if there are any other system configuration issues we should be
> looking at. As soon as we identify the source of the problem, I'll be happy
> to share it with the list.
>
> FWIW, when I was testing, there were a couple of instances where I
> accidentally loaded files with more than 1,000 records, and, even though it
> took some time, the processing ultimately completed. However, I don't recall
> now what kind of matching I was doing.
>
> I'm sorry I couldn't be more help on this one!
>
> Kathy
>
> --
> Kathy Lussier
> Project Coordinator
> Massachusetts Library Network Cooperative
> (508) 756-0172
> (508) 755-3721 (fax)
> klussier at masslnc.org
> Twitter: http://www.twitter.com/kmlussier
>
> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>> Hi Kathy,
>>
>> I'm cross-posting this question to the Evergreen Catalogers list.
>>
>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>> But I'm having problems when a file is large, more than 200 bibs. The
>> load screen seems to freeze at "Processing. 100". It never gets out of
>> the processing stage, and the bibs don't get loaded. I have several
>> large files (1000+) of OneClickDigital bib records that I need to load
>> through the client instead of through the database, since I need the
>> overlay feature to add URLs for each subscribing library. Are you
>> experiencing any difficulty when loading a large file?
>>
>> It seems like my only option is to divide the files into smaller
>> chunks, but that makes loading pretty labor intensive. I'm hoping
>> there's a fix for our loading logjam.
>>
>> Thanks,
>>
>> Mary
>>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 10 May 2012 11:04:37 -0400
> From: "Mary Llewellyn"<mllewell at biblio.org>
> To: "'Evergreen Community Catalogers'"
> 	<evergreen-catalogers at list.evergreen-ils.org>, 	"'Kathy Lussier'"
> 	<klussier at masslnc.org>
> Subject: Re: [Evergreen-catalogers] Vandelay and large files
> Message-ID:<003701cd2ebe$3777bb70$a6673250$@biblio.org>
> Content-Type: text/plain;	charset="us-ascii"
>
> Well, it was  a good news/bad news result. The good news was that with the
> leaner, meaner record match set (based solely on tag 001), the loader
> processed bibs beyond 100, and did it rapidly. The bad news is the file had
> 1972 bibs, and the processing screen froze at 1000. All the bibs made it to
> the queue, but none loaded.
>
> Mary
>
>
> -----Original Message-----
> From: evergreen-catalogers-bounces at list.evergreen-ils.org
> [mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf Of
> Mary Llewellyn
> Sent: Wednesday, May 09, 2012 4:10 PM
> To: 'Kathy Lussier'
> Cc: 'Evergreen Community Catalogers'
> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>
> Thanks, Kathy. I'll set up another matchpoint just for these OneClickDigital
> (or, as I like to call them, OCD) records, since they should have the same
> 001, so I don't have to worry about matching on 020.
>
> Wish me luck!
>
> Mary
>
>
> -----Original Message-----
> From: Kathy Lussier [mailto:klussier at masslnc.org]
> Sent: Wednesday, May 09, 2012 3:36 PM
> To: Mary Llewellyn
> Cc: Evergreen Community Catalogers
> Subject: Re: Vandelay and large files
>
> Hi Mary,
>
> I believe the MassLNC consortium that is live on Evergreen is typically
> loading files with fewer than 100 records. From what I understand, they are
> not having problems with load times.
>
> We do have another consortium, though, that is experiencing problems with
> load times for any size file if they are using a match set that matches on
> the 020, 022 or 024 field. I don't know if it's related, but, during the
> development, we had asked for some normalization on these fields (the 020
> field in particular) so that incoming records could match when there is pbk
> or other trailing text at the end of the field.
>
> Since this problem is happening in one Evergreen instance but not the other,
> we're looking at the possibility that we may be missing an index.
> I don't know if there are any other system configuration issues we should be
> looking at. As soon as we identify the source of the problem, I'll be happy
> to share it with the list.
>
> FWIW, when I was testing, there were a couple of instances where I
> accidentally loaded files with more than 1,000 records, and, even though it
> took some time, the processing ultimately completed. However, I don't recall
> now what kind of matching I was doing.
>
> I'm sorry I couldn't be more help on this one!
>
> Kathy
>
> --
> Kathy Lussier
> Project Coordinator
> Massachusetts Library Network Cooperative
> (508) 756-0172
> (508) 755-3721 (fax)
> klussier at masslnc.org
> Twitter: http://www.twitter.com/kmlussier
>
> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>> Hi Kathy,
>>
>> I'm cross-posting this question to the Evergreen Catalogers list.
>>
>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>> But I'm having problems when a file is large, more than 200 bibs. The
>> load screen seems to freeze at "Processing. 100". It never gets out of
>> the processing stage, and the bibs don't get loaded. I have several
>> large files (1000+) of OneClickDigital bib records that I need to load
>> through the client instead of through the database, since I need the
>> overlay feature to add URLs for each subscribing library. Are you
>> experiencing any difficulty when loading a large file?
>>
>> It seems like my only option is to divide the files into smaller
>> chunks, but that makes loading pretty labor intensive. I'm hoping
>> there's a fix for our loading logjam.
>>
>> Thanks,
>>
>> Mary
>>
> _______________________________________________
> Evergreen-catalogers mailing list
> Evergreen-catalogers at list.evergreen-ils.org
> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers
>
>
>
> ------------------------------
>
> _______________________________________________
> Evergreen-catalogers mailing list
> Evergreen-catalogers at list.evergreen-ils.org
> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catalogers
>
>
> End of Evergreen-catalogers Digest, Vol 1, Issue 10
> ***************************************************

-- 

Laurie Kulik
Database Manager/Cataloger
Merrimack Valley Library Consortium
1600 Osgood St.
N. Andover, MA 01845
(978) 557-8204 (work)
(978) 557-8101 (fax)