[OPEN-ILS-DEV] Evergreen Service Watchdog

[Don McMorris]

Hi Steve,

A number of Evergreen users utilize Nagios in monitoring Evergreen
services.  A recent presentation/training I gave shows some details,
and is available at
http://svn.open-ils.org/trac/ILS-Contrib/browser/ESI-Examples/trunk/docs/Presentations/Massachusetts%20Library%20Network%20Cooperative%20-%20May%202010/2.00_MassLNC_Training_May_2010%20Day_2.pdf
starting at page 4

For clark-kent.pl (as well as the hold targeter and fine generator),
we have a "check-lockfile.sh" script we often use - it can be found at
http://svn.open-ils.org/trac/ILS-Contrib/browser/ESI-Examples/sys/lib/nagios-plugins.
 In the case of clark-kent.pl - something that will often run
continuously - Nagios has a standard plugin to check processes.

For individual Evergreen services, a colleague recently scripted a
utilization check mechanism - it can be found at
http://esilibrary.com/~ldickens/eg-stats/.  This will log via syslog,
and adding a Nagios script to the syslog server can generate an alert
under certain conditions (usually high utilization or 0 running
processes).

Hope this helps!

--Don McMorris, Operations Manager
Equinox Software Inc


On Tue, May 18, 2010 at 3:00 PM, Steve Wills <steve.wills at lyrasis.org> wrote:
> Hi Guys,
>
> I discovered that openils.ingest had stopped running the other day and enough else was working so we didn't notice right away.  Clark-kent is another example of a service that can stop without causing the end of the world.  Has anyone written a watchdog to check that the fistful of services are all running and, hopefully, will attempt a restart and/or notify sysadmin if it find that services have stopped?
>
> Thanks,
> Stev3
> LYRASIS <--(they make me shout that.)
>