[OPEN-ILS-GENERAL] Here we grow again! Link checker functionality in Evergreen

[Duimovich, George]

>> *please* let's not develop an *integrated* link checker for Evergreen at all

Disagree -- there's no reason why link checker code couldn't be developed to be used (or easily adapted) to service both general purpose roles as well as "integrated" Staff Client functionality. What do you do for libraries who don't have direct server or easy database access? How else can cataloguers be empowered to service link errors, especially as we move towards increasing digital collections? 

Advantages of integration could include, for example, exposing batch editing and more effective reporting, etc.

I won't argue the point right now but "integration" needn't be incompatible with dis-integration, say encouraging the ability to generalize the service or allowing alternate solutions to the problem.

There is currently some sys admin oriented link checker code out there already for Evergreen that could be adapted (and improved) for general purpose usage (esp. for any database generated link listings, etc.)  For some basic working code, see our earlier submissions here:

http://markmail.org/message/v4cyszy33sgocqab and here: http://markmail.org/message/kgbpzgg25cm6fqcs  (with maybe some additional enhancements from Liam's work to be submitted by us later). 

But a good point to reinforce - it's a good design goal to ensure that any core link checker code be adaptable for more arbitrary purposes. 

Thanks for the feedback!

George

George Duimovich
NRCan Library / Bibliothèque de RNCan



-----Original Message-----
From: open-ils-general-bounces at list.georgialibraries.org [mailto:open-ils-general-bounces at list.georgialibraries.org] On Behalf Of Paul Hoffman
Sent: May 11, 2012 15:14
To: open-ils-general at list.georgialibraries.org; open-ils-dev at list.georgialibraries.org
Subject: [OPEN-ILS-GENERAL] ***SPAM*** Re: Here we grow again! Link checker functionality in Evergreen

On Fri, May 11, 2012 at 02:55:42PM -0400, Suzannah Lipscomb wrote:
> Equinox Software, Inc. is excited to announce the development of link 
> checker functionality in Evergreen.

Uh-oh!

> Evergreen currently has no built-in mechanism for verifying the 
> validity of URLs stored in MARC records. The ability to verify URLs 
> will be of particular benefit to locations with large electronic 
> resource collections. The requirements for this project are being 
> developed in partnership with NRCan Library and Statistics Canada 
> Library. The technical specifications for this project will be shared 
> with the Evergreen Community once they are ready. Equinox developers 
> estimate that coding will be completed no later than the end of the 
> third quarter of 2012.

As someone who has had to deal with poorly written link checkers and the havoc they wreak, I sincerely hope that one of the requirements will be
*politeness* -- specifically, a throttling mechanism that keeps the link checker from hammering away at a server that happens to serve a large number of resources linked to from an Evergreen catalog.  (This is actually quite simple in most cases: just shuffle the list of URLs that you check, and don't check too fast.  If you have 860,400 links in a catalog you can check ten per second and still finish in 24 hours, but you'd better make sure that those ten per second aren't all requested from the same server!)

And -- though it's not clear if this is what's intended or not --
*please* let's not develop an *integrated* link checker for Evergreen at all.  A link checker is properly a separate special-purpose tool with a simple, well-defined interface that allows it to be used by any number of applications; it shouldn't be all tangled up in an ILS.  There might even be a perfectly good link checker that fits the bill now -- I don't know, as I haven't looked into the matter very closely.

OK, I'll get down off my soap box now.  :-)

Paul.

--
Paul Hoffman <paul at flo.org>
Systems Librarian
Fenway Libraries Online
c/o Wentworth Institute of Technology
550 Huntington Ave.
Boston, MA 02115
(617) 445-2914
(617) 442-2384 (FLO main number)