[OPEN-ILS-GENERAL] ***SPAM*** Re: Here we grow again! Link checker functionality in Evergreen

[Lazar, Alexey Vladimirovich]

I would be surprized if there isn't a link checker already out there that could meet the needs.  Probably thousands have been written by now.  For example, Debian has a package: http://linux.die.net/man/1/linkchecker.

That said, sometimes it just makes sense to write one that very specifically meets the exact needs.  We had a home-brewed one when I worked at MNSU.  It worked very well for our specific needs, did exactly what we needed and nothing else.  From system administration perspective, there was no additional package and dependency bloat, but from developer perspective, there was certainly some code bloat and we were responsible for maintenance and bugs.

As far as politeness, I would assume this would be a matter of local policy and configuration.

Alexey Lazar
PALS
Information System Developer and Integrator
507-389-2907
http://www.mnpals.org/

On May 14, 2012, at 10:45 , Paul Hoffman wrote:

> I had written:
>> *please* let's not develop an *integrated* link checker for Evergreen at all
> 
> On Fri, May 11, 2012 at 06:17:21PM -0400, Duimovich, George wrote:
>> Disagree -- there's no reason why link checker code couldn't be 
>> developed to be used (or easily adapted) to service both general 
>> purpose roles as well as "integrated" Staff Client functionality. What 
>> do you do for libraries who don't have direct server or easy database 
>> access? How else can cataloguers be empowered to service link errors, 
>> especially as we move towards increasing digital collections? 
> 
> It sounds like we actually agree -- I would like to see Evergreen use an 
> external link checker, but the user shouldn't be able to tell that it's 
> external.
> 
>> Advantages of integration could include, for example, exposing batch 
>> editing and more effective reporting, etc.
>> 
>> I won't argue the point right now but "integration" needn't be 
>> incompatible with dis-integration, say encouraging the ability to 
>> generalize the service or allowing alternate solutions to the problem.
> 
> Exactly.
> 
>> There is currently some sys admin oriented link checker code out there 
>> already for Evergreen that could be adapted (and improved) for general 
>> purpose usage (esp. for any database generated link listings, etc.)  
>> For some basic working code, see our earlier submissions here:
>> 
>> http://markmail.org/message/v4cyszy33sgocqab and here: 
>> http://markmail.org/message/kgbpzgg25cm6fqcs  (with maybe some 
>> additional enhancements from Liam's work to be submitted by us later). 
> 
> Thanks for the pointers.
> 
> Paul.
> 
>> -----Original Message-----
>> From: open-ils-general-bounces at list.georgialibraries.org [mailto:open-ils-general-bounces at list.georgialibraries.org] On Behalf Of Paul Hoffman
>> Sent: May 11, 2012 15:14
>> To: open-ils-general at list.georgialibraries.org; open-ils-dev at list.georgialibraries.org
>> Subject: [OPEN-ILS-GENERAL] ***SPAM*** Re: Here we grow again! Link checker functionality in Evergreen
>> 
>> On Fri, May 11, 2012 at 02:55:42PM -0400, Suzannah Lipscomb wrote:
>>> Equinox Software, Inc. is excited to announce the development of link 
>>> checker functionality in Evergreen.
>> 
>> Uh-oh!
>> 
>>> Evergreen currently has no built-in mechanism for verifying the 
>>> validity of URLs stored in MARC records. The ability to verify URLs 
>>> will be of particular benefit to locations with large electronic 
>>> resource collections. The requirements for this project are being 
>>> developed in partnership with NRCan Library and Statistics Canada 
>>> Library. The technical specifications for this project will be shared 
>>> with the Evergreen Community once they are ready. Equinox developers 
>>> estimate that coding will be completed no later than the end of the 
>>> third quarter of 2012.
>> 
>> As someone who has had to deal with poorly written link checkers and the havoc they wreak, I sincerely hope that one of the requirements will be
>> *politeness* -- specifically, a throttling mechanism that keeps the link checker from hammering away at a server that happens to serve a large number of resources linked to from an Evergreen catalog.  (This is actually quite simple in most cases: just shuffle the list of URLs that you check, and don't check too fast.  If you have 860,400 links in a catalog you can check ten per second and still finish in 24 hours, but you'd better make sure that those ten per second aren't all requested from the same server!)
>> 
>> And -- though it's not clear if this is what's intended or not --
>> *please* let's not develop an *integrated* link checker for Evergreen at all.  A link checker is properly a separate special-purpose tool with a simple, well-defined interface that allows it to be used by any number of applications; it shouldn't be all tangled up in an ILS.  There might even be a perfectly good link checker that fits the bill now -- I don't know, as I haven't looked into the matter very closely.
>> 
>> OK, I'll get down off my soap box now.  :-)
>> 
>> Paul.
>> 
>> --
>> Paul Hoffman <paul at flo.org>
>> Systems Librarian
>> Fenway Libraries Online
>> c/o Wentworth Institute of Technology
>> 550 Huntington Ave.
>> Boston, MA 02115
>> (617) 445-2914
>> (617) 442-2384 (FLO main number)
> 
> -- 
> Paul Hoffman <paul at flo.org>
> Systems Librarian
> Fenway Libraries Online
> c/o Wentworth Institute of Technology
> 550 Huntington Ave.
> Boston, MA 02115
> (617) 445-2914
> (617) 442-2384 (FLO main number)