[OPEN-ILS-DEV] Notification system design

[Bill Erickson]

I'm going through my email and just want to drop a quick note...

Hold notification emails and overdue notice emails both use very similar
templating logic - basically, just some home-spun search/replace regexes.
The actual support code for each was developed at different times, though,
which gives them a very ad-hoc feel and means there is some code duplication
going on.  Unifying these interfaces, adding basic delay functionality, and
standardizing the template interface (template toolkit?) would be a worthy
phase 1 goal, in my opinion, and I think those alone are enough to justify
the effort.  Adding extended functionality would then be a nice bonus.

back to email...

-bill



On 12/29/06, Mike Rylander <mrylander at gmail.com> wrote:
>
> On 12/29/06, Nathan Eady <eady at galion.lib.oh.us> wrote:
> > Mike Rylander wrote:
> [snip]
>
> > > The first and second there are fairly simple.
> > Conceptually simple, anyway.
>
> And practically.  The reporting system, the billing system and the
> hold targeting system all already do something like this, but with
> enough specialized logic required to warrant their own queues.
>
> >
> > > Event aggregation is a bit more tricky.  It isn't technically hard,
> > > but it's also not easy to make it flexible when there's over 100
> > > object types, most linked in some way, all with useful data hidden
> > > behind those links.  If we were building _just_ a hold event queue,
> > > then it's easy, but if we want it to be generalized for patron events,
> > > and circulation events, and payment events and who knows what else,
> > > then it's less simple.  This could potentially be simplified at the DB
> > > level by judicious use of views (among other techniques), so it may
> > > not be as bad as I fear.
> >
> > In some ways it is tempting to push this in the OO direction and
> > have each type (or "class") of event provide an aggregator function.
>
> I think going that way may be a bit too complex.  Depending on the
> first-class information we keep on the event (type, target data,
> target user, addition time, trigger time, a couple generic qualifiers
> defined by the type, etc) it can be a simple set of filters that
> translate directly to an SQL WHERE clause.  The filters are stored as
> a set of operator-value pairs attached to the type and qualifier set,
> and the event firing mechanism does the aggregation directly where
> instructed.
>
> > On the other hand, if we viewed _all_ notifications as a single
> > class or type of event, with those other distinctions just being
> > some of the auxilliary data, then it might seem manageable again.
> > The aggregator function for the notifications event class would
> > be complicated, but at least the complexity would be all bundled
> > up inside that function, and other types of events with different
> > needs (whether simpler or more complex) likewise.
>
> Right.  And even that could be served well by cutting it up into a
> front-line type specific processor function which simply delegates to
> chunks of code that do the actual processing.
>
> > > Anyway, just wanted to get some blue-sky thoughts out there.  Is
> > > this generally what you guys have been seeing in your heads?
> >
> > Generally.
>
> Well, that's a start! ;)
>
> [snip]
>
> > >> It seems reasonable that "pending" notifications (i.e., those that
> > >> have been triggered or queued but not yet sent) should inhabit a
> > >> table in the database.
> > >
> > > Oh, of course.  I meant in terms of historical reporting.
> >
> > Oh, that, yes.  Some kind of record of what's been done is always
> > wanted.  You can do that with log files, but keeping it in a database
> > makes it easier to pull statistics from later.
> >
> > Surely, though, you must have something like that already.  But
> > not a unified event infrastructure.  Does that mean what you're
> > thinking of is replacing (some of) what you have now with a more
> > general infrastructure that can handle more than what it already
> > does?
>
> We do.  There is a reporting system built on top of the data, along
> with some specialized views to simplify certain data extraction.  It
> utilizes the same IDL file that the rest of the system uses for
> generating object classes, so it already knows about all the data in
> the main storage engine.  This would automatically extend to events,
> past and future, if we retain all of them (and layer on some views to
> split them based on, say, type).
>
> I was just pointing in the direction of "do we retain all history by
> marking events 'complete'" or "do we remove events after completion."
> The former is, in some regards, simpler and gains us more in terms of
> reporting.  To me, the choice is self-evident ... but I wanted to
> bring up the options.
>
> I imagine the event system would absorb most of the current
> notification logic, but nothing other than that.
>
> > Perhaps each type of event (e.g. a circulation event (which might be
> > a checkin, checkout, renewal, or whatever along those lines), versus
>
> None of that would use this new infrastructure.  There's already
> specialized circulation logic, and that must all happen real-time.
> Sorry if I gave the impression that we'd want to move that kind of
> stuff.
>
> Just to clarify, nothing that requires real-time or transactional
> (ACID) semantics can ever use this.  This new infrastructure would
> absorb, at most, the current hold and overdue (including billing)
> notification.  Any delayable events could (hopefully) be built on this
> as well.
>
> When I mentioned circulation based events I meant events created by a
> circulation, not representing them.  Imagine an offer-to-review event
> that is generated on every 1,000th circ, or maybe on every circ of new
> books, or whatever.  The circ would generate an event, and the event
> processor would create and send the needed information (authorization
> token, email of the offer, etc).
>
> > Conceptually, I could be persuaded to like the idea of having the
> > same events subsystem handle both keeping data around for future
> > reporting *and* hanging onto notifications until they are ready
> > to be "sent" (in whatever fashion), and other... stuff.  It means
> > once you learn how one (sub)system works, you then already know
> > stuff about how the other stuff works.
>
> Generally, that's the idea.  Creating a queuing and triggering
> infrastructure for every non-real-time event type seems like a
> maintenance nightmare.  Certain subsystems need their own, but any
> that are acceptable with a delay, or even require one, should be able
> to use the same basic infrastructure ... that's where I'm hoping all
> this heads.  ;)
>
> --
> Mike Rylander
> mrylander at gmail.com
> GPLS -- PINES Development
> Database Developer
> http://open-ils.org
>



-- 
Bill Erickson
PINES Systems Developer
Georgia Public Library Service
billserickson at gmail.com
http://open-ils.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20070102/3dff0d2b/attachment.html