[OPEN-ILS-DEV] Slow Vandelay uploads

[Kathy Lussier]

Hi all,

We've been experiencing slow Vandelay uploads on one of our Evergreen
instances (C/W MARS), and I'm hoping somebody can steer us in the right
direction for identifying the cause of the problem.

The problems are happening in the following environment:
Evergreen version: 2.2 beta2
PostgreSQL version: 9.1
OpenSRF: 2.1.0 RC1
Distribution: DebianSqueeze

The slow uploads primarily occur when using match sets that match on the
020, 022 or 024 tags. From previous discussions I had when the Vandelay
match sets were being developed, I believe these are the three fields where
there is some left-anchored truncation so that a bib record with trailing
pbk text in the 020 field would match on records that do not contain this
text. At first glance, it appears as if this truncation has added time to
the upload process.

However, we are also seeing a significant difference for one of our
Evergreen instances that is not seen on the other two, leading me to believe
that there may also be something in the system configuration that could be
contributing to the slow upload times. 

As an example, using a match set that matches on the 020a, 022a, 024a, or
028a fields, I uploaded a file with 100 records on a 2.2 RC1 system
yesterday. The upload took 7 1/2 minutes. Although not ideal, this upload
time doesn't raise any red flags. 

I tried uploading the same file this morning on the C/W MARS server using
the same fields as match points. The upload took 21 1/2 minutes. A few
months ago, Dan and Mike had suggested the problem might be due to a missing
index. C/W MARS added an index for 020a in metabib.real_full_rec last week,
but the upload times are still slow. The previously-mentioned file of 100
records took about 13 minutes to load when only matching on the 020 field in
the C/W MARS environment.

Is there anything else we should be looking at to improve the upload times? 

I don't know if it is related, but we are seeing these slower uploads in a
brick environment. The other servers where I have seen faster upload times
are not in a brick environment. Could this be contributing to the problem?

Thank you in advance for any insight you can offer on the problem!

Kathy


-------------------------------------------------------------
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 756-0172
(508) 755-3721 (fax)
klussier at masslnc.org
IM: kmlussier (AOL & Yahoo)
Twitter: http://www.twitter.com/kmlussier