[OPEN-ILS-DEV] Notification system design

[Mike Rylander]

On 12/29/06, Nathan Eady <eady at galion.lib.oh.us> wrote:
> Mike Rylander wrote:
[snip]

> > The first and second there are fairly simple.
> Conceptually simple, anyway.

And practically.  The reporting system, the billing system and the
hold targeting system all already do something like this, but with
enough specialized logic required to warrant their own queues.

>
> > Event aggregation is a bit more tricky.  It isn't technically hard,
> > but it's also not easy to make it flexible when there's over 100
> > object types, most linked in some way, all with useful data hidden
> > behind those links.  If we were building _just_ a hold event queue,
> > then it's easy, but if we want it to be generalized for patron events,
> > and circulation events, and payment events and who knows what else,
> > then it's less simple.  This could potentially be simplified at the DB
> > level by judicious use of views (among other techniques), so it may
> > not be as bad as I fear.
>
> In some ways it is tempting to push this in the OO direction and
> have each type (or "class") of event provide an aggregator function.

I think going that way may be a bit too complex.  Depending on the
first-class information we keep on the event (type, target data,
target user, addition time, trigger time, a couple generic qualifiers
defined by the type, etc) it can be a simple set of filters that
translate directly to an SQL WHERE clause.  The filters are stored as
a set of operator-value pairs attached to the type and qualifier set,
and the event firing mechanism does the aggregation directly where
instructed.

> On the other hand, if we viewed _all_ notifications as a single
> class or type of event, with those other distinctions just being
> some of the auxilliary data, then it might seem manageable again.
> The aggregator function for the notifications event class would
> be complicated, but at least the complexity would be all bundled
> up inside that function, and other types of events with different
> needs (whether simpler or more complex) likewise.

Right.  And even that could be served well by cutting it up into a
front-line type specific processor function which simply delegates to
chunks of code that do the actual processing.

> > Anyway, just wanted to get some blue-sky thoughts out there.  Is
> > this generally what you guys have been seeing in your heads?
>
> Generally.

Well, that's a start! ;)

[snip]

> >> It seems reasonable that "pending" notifications (i.e., those that
> >> have been triggered or queued but not yet sent) should inhabit a
> >> table in the database.
> >
> > Oh, of course.  I meant in terms of historical reporting.
>
> Oh, that, yes.  Some kind of record of what's been done is always
> wanted.  You can do that with log files, but keeping it in a database
> makes it easier to pull statistics from later.
>
> Surely, though, you must have something like that already.  But
> not a unified event infrastructure.  Does that mean what you're
> thinking of is replacing (some of) what you have now with a more
> general infrastructure that can handle more than what it already
> does?

We do.  There is a reporting system built on top of the data, along
with some specialized views to simplify certain data extraction.  It
utilizes the same IDL file that the rest of the system uses for
generating object classes, so it already knows about all the data in
the main storage engine.  This would automatically extend to events,
past and future, if we retain all of them (and layer on some views to
split them based on, say, type).

I was just pointing in the direction of "do we retain all history by
marking events 'complete'" or "do we remove events after completion."
The former is, in some regards, simpler and gains us more in terms of
reporting.  To me, the choice is self-evident ... but I wanted to
bring up the options.

I imagine the event system would absorb most of the current
notification logic, but nothing other than that.

> Perhaps each type of event (e.g. a circulation event (which might be
> a checkin, checkout, renewal, or whatever along those lines), versus

None of that would use this new infrastructure.  There's already
specialized circulation logic, and that must all happen real-time.
Sorry if I gave the impression that we'd want to move that kind of
stuff.

Just to clarify, nothing that requires real-time or transactional
(ACID) semantics can ever use this.  This new infrastructure would
absorb, at most, the current hold and overdue (including billing)
notification.  Any delayable events could (hopefully) be built on this
as well.

When I mentioned circulation based events I meant events created by a
circulation, not representing them.  Imagine an offer-to-review event
that is generated on every 1,000th circ, or maybe on every circ of new
books, or whatever.  The circ would generate an event, and the event
processor would create and send the needed information (authorization
token, email of the offer, etc).

> Conceptually, I could be persuaded to like the idea of having the
> same events subsystem handle both keeping data around for future
> reporting *and* hanging onto notifications until they are ready
> to be "sent" (in whatever fashion), and other... stuff.  It means
> once you learn how one (sub)system works, you then already know
> stuff about how the other stuff works.

Generally, that's the idea.  Creating a queuing and triggering
infrastructure for every non-real-time event type seems like a
maintenance nightmare.  Certain subsystems need their own, but any
that are acceptable with a delay, or even require one, should be able
to use the same basic infrastructure ... that's where I'm hoping all
this heads.  ;)

-- 
Mike Rylander
mrylander at gmail.com
GPLS -- PINES Development
Database Developer
http://open-ils.org