[Evergreen-catalogers] Vandelay and large files

Mon May 14 13:11:26 EDT 2012

Janet,

We do the Evergreen MARC File Upload first, match on our Record Match 
Set (020, 022, 024, 028) and then upload the appropriate file. Second, 
we look at the queue, check off all matches, check off View MARC and 
uncheck records with more than one or no match. Then choose Import 
Selected Records, Merge Profile and Merge on Best Match; Import. Limit 
to Non-imported Records and we check each of these which are considered 
"problem" records (merges, no match, etc.).

Hope this makes sense!

I realize your workload is much bigger than ours so checking single 
records is not practical.
Laurie

On 5/14/2012 12:34 PM, Janet Schrader wrote:
> I have been working with the same file using various permutations of loading records. Last week Sue installed the missing index and re-indexed all records. There was no noticeable improvement after the re-indexing. See text below between the>>>  <<<<
>
> On Alpha3 I loaded a file of 500 records for online resources, matching on a 999 tag I inserted into records using MARCedit so these would overlay and insert additional 856 fields. I was able to load 500 records in<>5 min.
>
> Laurie, I'm not sure what you mean by doing the upload separately. When do you choose the Record Match Set? If I do not chose anything and just do the upload the file loads in a>15 seconds but there is nothing to show which records match. At this point I cannot choose a Record Match Set, only a Merge/Overlay profile.
>
> If I choose my Record Match Set to match only on ISBN then the file still takes the<20 minutes to process.
>
> I believe the problem has got to be with the 020 field. So-- where do we go from here?
>
> I loaded the same file of 159 records twice using the same Record Match Set with or without quality metrics for encoding levels. I used the same Overlay/Merge profile.
>
> Match on 020 with quality metrics for encoding level:
> File: bcat0312_159 recd 2
> Processing: 22 minutes
> Load:>2 min.
>
> Match on 020 without quality metrics (only thing in profile is match on 020):
> File: bcat0312_159 rec no enc lvl
> Processing: 21 minutes
> Load:>2 min.
>
> There was no difference in the speed of the processing from yesterday before re-indexing. There was no difference because of the quality metrics. The only difference was without quality metrics, records with lower encoding levels were not rejected.
>
> I loaded this same file using the profile for online resources (match on tag I insert in MARC record). Since this file has nothing to match on the file processed AND loaded in 1 min. 30 seconds.
> <<<<<<<<<<<<<<<
>
> Janet
>
> Janet Schrader
> C/W MARS Inc.
> 67 Millbrook Street, Suite 201
> Worcester, MA 01606
> tel: 508-755-3323 ext. 25
> fax: 508-757-7801
> jschrader at cwmars.org
>
>
>
> -----Original Message-----
> From: evergreen-catalogers-bounces at list.evergreen-ils.org [mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf Of Laurie
> Sent: Thursday, May 10, 2012 1:25 PM
> To: evergreen-catalogers at list.evergreen-ils.org; Kathy Lussier; mllewell at biblio.org; Regina
> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>
> Hi Mary and Kathy,
>
> At MVLC we have been doing uploads separately as well as doing the process together. When doing them separately the import/export process does seem to do larger queues more quickly, while together it does bog down. I think that is what happens with the CWMars consortium because they do much larger queues than we do. I have done separate queues of under 200 records that did process quickly, so that might be helpful.
>
> What this means is that we do the upload first and then the import choosing whichever Record Import Action is most practical for your needs.
>
> Hope this helps,
> Laurie
>
>
>
> On 5/10/2012 12:00 PM,
> evergreen-catalogers-request at list.evergreen-ils.org wrote:
>> Send Evergreen-catalogers mailing list submissions to
>> 	evergreen-catalogers at list.evergreen-ils.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	
>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catal
>> ogers
>>
>> or, via email, send a message with subject or body 'help' to
>> 	evergreen-catalogers-request at list.evergreen-ils.org
>>
>> You can reach the person managing the list at
>> 	evergreen-catalogers-owner at list.evergreen-ils.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Evergreen-catalogers digest..."
>>
>>
>> Today's Topics:
>>
>>      1. Vandelay and large files (Mary Llewellyn)
>>      2. Re: Vandelay and large files (Kathy Lussier)
>>      3. Re: Vandelay and large files (Mary Llewellyn)
>>      4. Re: Vandelay and large files (Mary Llewellyn)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 9 May 2012 14:21:46 -0400
>> From: "Mary Llewellyn"<mllewell at biblio.org>
>> To: "'Kathy Lussier'"<klussier at masslnc.org>, "Evergreen Community
>> 	Catalogers"	<evergreen-catalogers at list.evergreen-ils.org>
>> Subject: [Evergreen-catalogers] Vandelay and large files
>> Message-ID:<00d501cd2e10$97b41b90$c71c52b0$@biblio.org>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi Kathy,
>>
>>
>>
>> I'm cross-posting this question to the Evergreen Catalogers list.
>>
>>
>>
>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>> But I'm having problems when a file is large, more than 200 bibs. The
>> load screen seems to freeze at "Processing. 100". It never gets out of
>> the processing stage, and the bibs don't get loaded. I have several
>> large files (1000+)  of OneClickDigital bib records that I need to
>> load through the client instead of through the database, since I need
>> the overlay feature to add URLs for each subscribing library. Are you
>> experiencing any difficulty when loading a large file?
>>
>>
>>
>> It seems like my only option is to divide the files into smaller
>> chunks, but that makes loading pretty labor intensive. I'm hoping
>> there's a fix for our loading logjam.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Mary
>>
>>
>>
>> -------------- next part -------------- An HTML attachment was
>> scrubbed...
>> URL:<http://list.evergreen-ils.org/pipermail/evergreen-catalogers/atta
>> chments/20120509/f35f5064/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 09 May 2012 15:36:15 -0400
>> From: Kathy Lussier<klussier at masslnc.org>
>> To: Mary Llewellyn<mllewell at biblio.org>
>> Cc: Evergreen Community Catalogers
>> 	<evergreen-catalogers at list.evergreen-ils.org>
>> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>> Message-ID:<4FAAC72F.6090609 at masslnc.org>
>> Content-Type: text/plain; charset=windows-1252; format=flowed
>>
>> Hi Mary,
>>
>> I believe the MassLNC consortium that is live on Evergreen is
>> typically loading files with fewer than 100 records. From what I
>> understand, they are not having problems with load times.
>>
>> We do have another consortium, though, that is experiencing problems
>> with load times for any size file if they are using a match set that
>> matches on the 020, 022 or 024 field. I don't know if it's related,
>> but, during the development, we had asked for some normalization on
>> these fields (the 020 field in particular) so that incoming records
>> could match when there is pbk or other trailing text at the end of the field.
>>
>> Since this problem is happening in one Evergreen instance but not the
>> other, we're looking at the possibility that we may be missing an index.
>> I don't know if there are any other system configuration issues we
>> should be looking at. As soon as we identify the source of the
>> problem, I'll be happy to share it with the list.
>>
>> FWIW, when I was testing, there were a couple of instances where I
>> accidentally loaded files with more than 1,000 records, and, even
>> though it took some time, the processing ultimately completed.
>> However, I don't recall now what kind of matching I was doing.
>>
>> I'm sorry I couldn't be more help on this one!
>>
>> Kathy
>>
>> --
>> Kathy Lussier
>> Project Coordinator
>> Massachusetts Library Network Cooperative
>> (508) 756-0172
>> (508) 755-3721 (fax)
>> klussier at masslnc.org
>> Twitter: http://www.twitter.com/kmlussier
>>
>> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>>> Hi Kathy,
>>>
>>> I?m cross-posting this question to the Evergreen Catalogers list.
>>>
>>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>>> But I?m having problems when a file is large, more than 200 bibs. The
>>> load screen seems to freeze at ?Processing? 100?. It never gets out
>>> of the processing stage, and the bibs don?t get loaded. I have
>>> several large files (1000+) of OneClickDigital bib records that I
>>> need to load through the client instead of through the database,
>>> since I need the overlay feature to add URLs for each subscribing
>>> library. Are you experiencing any difficulty when loading a large file?
>>>
>>> It seems like my only option is to divide the files into smaller
>>> chunks, but that makes loading pretty labor intensive. I?m hoping
>>> there?s a fix for our loading logjam.
>>>
>>> Thanks,
>>>
>>> Mary
>>>
>> ------------------------------
>>
>> Message: 3
>> Date: Wed, 9 May 2012 16:10:22 -0400
>> From: "Mary Llewellyn"<mllewell at biblio.org>
>> To: "'Kathy Lussier'"<klussier at masslnc.org>
>> Cc: 'Evergreen Community Catalogers'
>> 	<evergreen-catalogers at list.evergreen-ils.org>
>> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>> Message-ID:<00f001cd2e1f$c3bd4f40$4b37edc0$@biblio.org>
>> Content-Type: text/plain;	charset="us-ascii"
>>
>> Thanks, Kathy. I'll set up another matchpoint just for these
>> OneClickDigital (or, as I like to call them, OCD) records, since they
>> should have the same 001, so I don't have to worry about matching on 020.
>>
>> Wish me luck!
>>
>> Mary
>>
>>
>> -----Original Message-----
>> From: Kathy Lussier [mailto:klussier at masslnc.org]
>> Sent: Wednesday, May 09, 2012 3:36 PM
>> To: Mary Llewellyn
>> Cc: Evergreen Community Catalogers
>> Subject: Re: Vandelay and large files
>>
>> Hi Mary,
>>
>> I believe the MassLNC consortium that is live on Evergreen is
>> typically loading files with fewer than 100 records. From what I
>> understand, they are not having problems with load times.
>>
>> We do have another consortium, though, that is experiencing problems
>> with load times for any size file if they are using a match set that
>> matches on the 020, 022 or 024 field. I don't know if it's related,
>> but, during the development, we had asked for some normalization on
>> these fields (the 020 field in particular) so that incoming records
>> could match when there is pbk or other trailing text at the end of the field.
>>
>> Since this problem is happening in one Evergreen instance but not the
>> other, we're looking at the possibility that we may be missing an index.
>> I don't know if there are any other system configuration issues we
>> should be looking at. As soon as we identify the source of the
>> problem, I'll be happy to share it with the list.
>>
>> FWIW, when I was testing, there were a couple of instances where I
>> accidentally loaded files with more than 1,000 records, and, even
>> though it took some time, the processing ultimately completed.
>> However, I don't recall now what kind of matching I was doing.
>>
>> I'm sorry I couldn't be more help on this one!
>>
>> Kathy
>>
>> --
>> Kathy Lussier
>> Project Coordinator
>> Massachusetts Library Network Cooperative
>> (508) 756-0172
>> (508) 755-3721 (fax)
>> klussier at masslnc.org
>> Twitter: http://www.twitter.com/kmlussier
>>
>> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>>> Hi Kathy,
>>>
>>> I'm cross-posting this question to the Evergreen Catalogers list.
>>>
>>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>>> But I'm having problems when a file is large, more than 200 bibs. The
>>> load screen seems to freeze at "Processing. 100". It never gets out
>>> of the processing stage, and the bibs don't get loaded. I have
>>> several large files (1000+) of OneClickDigital bib records that I
>>> need to load through the client instead of through the database,
>>> since I need the overlay feature to add URLs for each subscribing
>>> library. Are you experiencing any difficulty when loading a large file?
>>>
>>> It seems like my only option is to divide the files into smaller
>>> chunks, but that makes loading pretty labor intensive. I'm hoping
>>> there's a fix for our loading logjam.
>>>
>>> Thanks,
>>>
>>> Mary
>>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Thu, 10 May 2012 11:04:37 -0400
>> From: "Mary Llewellyn"<mllewell at biblio.org>
>> To: "'Evergreen Community Catalogers'"
>> 	<evergreen-catalogers at list.evergreen-ils.org>, 	"'Kathy Lussier'"
>> 	<klussier at masslnc.org>
>> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>> Message-ID:<003701cd2ebe$3777bb70$a6673250$@biblio.org>
>> Content-Type: text/plain;	charset="us-ascii"
>>
>> Well, it was  a good news/bad news result. The good news was that with
>> the leaner, meaner record match set (based solely on tag 001), the
>> loader processed bibs beyond 100, and did it rapidly. The bad news is
>> the file had
>> 1972 bibs, and the processing screen froze at 1000. All the bibs made
>> it to the queue, but none loaded.
>>
>> Mary
>>
>>
>> -----Original Message-----
>> From: evergreen-catalogers-bounces at list.evergreen-ils.org
>> [mailto:evergreen-catalogers-bounces at list.evergreen-ils.org] On Behalf
>> Of Mary Llewellyn
>> Sent: Wednesday, May 09, 2012 4:10 PM
>> To: 'Kathy Lussier'
>> Cc: 'Evergreen Community Catalogers'
>> Subject: Re: [Evergreen-catalogers] Vandelay and large files
>>
>> Thanks, Kathy. I'll set up another matchpoint just for these
>> OneClickDigital (or, as I like to call them, OCD) records, since they
>> should have the same 001, so I don't have to worry about matching on 020.
>>
>> Wish me luck!
>>
>> Mary
>>
>>
>> -----Original Message-----
>> From: Kathy Lussier [mailto:klussier at masslnc.org]
>> Sent: Wednesday, May 09, 2012 3:36 PM
>> To: Mary Llewellyn
>> Cc: Evergreen Community Catalogers
>> Subject: Re: Vandelay and large files
>>
>> Hi Mary,
>>
>> I believe the MassLNC consortium that is live on Evergreen is
>> typically loading files with fewer than 100 records. From what I
>> understand, they are not having problems with load times.
>>
>> We do have another consortium, though, that is experiencing problems
>> with load times for any size file if they are using a match set that
>> matches on the 020, 022 or 024 field. I don't know if it's related,
>> but, during the development, we had asked for some normalization on
>> these fields (the 020 field in particular) so that incoming records
>> could match when there is pbk or other trailing text at the end of the field.
>>
>> Since this problem is happening in one Evergreen instance but not the
>> other, we're looking at the possibility that we may be missing an index.
>> I don't know if there are any other system configuration issues we
>> should be looking at. As soon as we identify the source of the
>> problem, I'll be happy to share it with the list.
>>
>> FWIW, when I was testing, there were a couple of instances where I
>> accidentally loaded files with more than 1,000 records, and, even
>> though it took some time, the processing ultimately completed.
>> However, I don't recall now what kind of matching I was doing.
>>
>> I'm sorry I couldn't be more help on this one!
>>
>> Kathy
>>
>> --
>> Kathy Lussier
>> Project Coordinator
>> Massachusetts Library Network Cooperative
>> (508) 756-0172
>> (508) 755-3721 (fax)
>> klussier at masslnc.org
>> Twitter: http://www.twitter.com/kmlussier
>>
>> On 5/9/2012 2:21 PM, Mary Llewellyn wrote:
>>> Hi Kathy,
>>>
>>> I'm cross-posting this question to the Evergreen Catalogers list.
>>>
>>> I love using the new Vandelay in our Evergreen 2.2-ish environment.
>>> But I'm having problems when a file is large, more than 200 bibs. The
>>> load screen seems to freeze at "Processing. 100". It never gets out
>>> of the processing stage, and the bibs don't get loaded. I have
>>> several large files (1000+) of OneClickDigital bib records that I
>>> need to load through the client instead of through the database,
>>> since I need the overlay feature to add URLs for each subscribing
>>> library. Are you experiencing any difficulty when loading a large file?
>>>
>>> It seems like my only option is to divide the files into smaller
>>> chunks, but that makes loading pretty labor intensive. I'm hoping
>>> there's a fix for our loading logjam.
>>>
>>> Thanks,
>>>
>>> Mary
>>>
>> _______________________________________________
>> Evergreen-catalogers mailing list
>> Evergreen-catalogers at list.evergreen-ils.org
>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catal
>> ogers
>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> Evergreen-catalogers mailing list
>> Evergreen-catalogers at list.evergreen-ils.org
>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-catal
>> ogers
>>
>>
>> End of Evergreen-catalogers Digest, Vol 1, Issue 10
>> ***************************************************

-- 

Laurie Kulik
Database Manager/Cataloger
Merrimack Valley Library Consortium
1600 Osgood St.
N. Andover, MA 01845
(978) 557-8204 (work)
(978) 557-8101 (fax)