[OPEN-ILS-DEV] Notification system design

[Nathan Eady]

Mike Rylander wrote:
> It seems we may have meandered off the path a bit into a field of
> specifics ...

Sorry, I didn't mean to imply that we should implement specific
cases at this point; I was mainly trying to suggest a general and
flexible framework that would be able to accommodate a range of
behaviors, but it seems you were already two steps ahead of me there.

> From the thoughts so far, it feels like we are all thinking of a
> similar subsystem.  Looking at it from the broadest possible view,
> outside an ILS perspective, here's what I see:
[snip]

That all sounds pretty reasonable to me.

> The first and second there are fairly simple.

Conceptually simple, anyway.

> Event aggregation is a bit more tricky.  It isn't technically hard,
> but it's also not easy to make it flexible when there's over 100
> object types, most linked in some way, all with useful data hidden
> behind those links.  If we were building _just_ a hold event queue,
> then it's easy, but if we want it to be generalized for patron events,
> and circulation events, and payment events and who knows what else,
> then it's less simple.  This could potentially be simplified at the DB
> level by judicious use of views (among other techniques), so it may
> not be as bad as I fear.

In some ways it is tempting to push this in the OO direction and
have each type (or "class") of event provide an aggregator function.

However, that could get complicated when you inevitably end up with
subtypes of events.  In the case of hold notifications, you might
potentially end up with two orthogonal hierarchies of classes (one
for hold notification, overdue warning, overdue notification,
biller, and so on and so forth, and the other for email notification,
postal notification, phone notification, yadda, yadda).  OO systems
with that level of sophistication can be complex and might be quite
more than is needed for an events queue, even a fairly flexible one.

So it might be better to go in another direction.

On the other hand, if we viewed _all_ notifications as a single
class or type of event, with those other distinctions just being
some of the auxilliary data, then it might seem manageable again.
The aggregator function for the notifications event class would
be complicated, but at least the complexity would be all bundled
up inside that function, and other types of events with different
needs (whether simpler or more complex) likewise.

I'm still not sure if that's the way to go though.  My experience
with stuff this complex is... limited.

> Anyway, just wanted to get some blue-sky thoughts out there.  Is
> this generally what you guys have been seeing in your heads?

Generally.

> Oh, sure.  No argument there.  There are currently scripts that
> generate data for postal notification, including aggregating several
> types of notifications into one block.  They spit out a simple XML
> file and are controlled by cron.  Bill can get into more of that when
> he returns... :)

Those details are not important to me at this time, except insofar
as they have an impact on the design of the general notification
system.  My main concern in bringing up postal notification was
that there needs to be a queueing system that can hold events until
they are called for, even if they are called for at unpredictable
intervals.  I think the design you discussed should be able to
handle that.

Again, you were a step ahead of me.

Something you had said in reply to Josh lead me to believe you
hadn't gotten this far yet.  Ah, here it was:

 > There are no design docs for the notification infrastructure
 > as yet.  This was nearly all Bill's stuff, so I don't know how
 > far we should go while he's not around to defend himself ;),
 > but a generalized notification service is something we've
 > discussed (and makes sense, if more than email notification
 > is desired).

Apparently when you said "something we've discussed" you
meant more than just "briefly discussed the need for".  I
read too much into the "no design docs... as yet" part.

>> It seems reasonable that "pending" notifications (i.e., those that
>> have been triggered or queued but not yet sent) should inhabit a
>> table in the database.
> 
> Oh, of course.  I meant in terms of historical reporting.  

Oh, that, yes.  Some kind of record of what's been done is always
wanted.  You can do that with log files, but keeping it in a database
makes it easier to pull statistics from later.

Surely, though, you must have something like that already.  But
not a unified event infrastructure.  Does that mean what you're
thinking of is replacing (some of) what you have now with a more
general infrastructure that can handle more than what it already
does?

> I hesitate to through away any data if I can avoid it, but even
 > notifications are personally identifiable data...

What to keep and what to throw away in such a situation perhaps
should be up to site policy, but if you throw away things like the
patron ID (because it points to personally identifiable information,
such as the patron registration, elsewhere in the database), then
you need to expressly keep anything you might actually _need_ that
it also points to.  So any time you throw away information, some code
needs to get a crack at grabbing what it needs from it first, yes?

This starts to step outside the bounds of notifications, but for
instance in the case of a "check out" transaction, you probably
want to be able to get access to various demographic data on the
patron (age, zipcode, maybe school district, possibly gender, or
whatever) later, in order to be able to look at it in the aggregate,
so if you throw out the patron ID you want to keep all that stuff.

Perhaps each type of event (e.g. a circulation event (which might be
a checkin, checkout, renewal, or whatever along those lines), versus
a notification event or some other thing) needs to have, in addition
to an aggregator, also a "sanitizer".  When would it run, though?
After some pre-defined interval?  Only if the system load is below
some threshhold?  Overnight processing?  (Are there libraries open
24-hours, that would not want to have a lot of stuff happening in
overnight processing?  Even a quick sanitizer function could add
up to a few minutes if it's called for every circ event for an
entire day, in a large library system.)  Obviously it wouldn't be
run on "incomplete" events, e.g., notifications that haven't been
sent yet.

Conceptually, I could be persuaded to like the idea of having the
same events subsystem handle both keeping data around for future
reporting *and* hanging onto notifications until they are ready
to be "sent" (in whatever fashion), and other... stuff.  It means
once you learn how one (sub)system works, you then already know
stuff about how the other stuff works.

It does however raise the question of whether to aggregate before
or after sanitizing (assuming I haven't wandered off and gotten
lost again in my musings above).  Circ events perhaps should be
aggregated before sanitizing, so that you know that a single
patron checked out thirteen items, as opposed to only knowing that
all thirteen items were checked out by patron(s) from a certain
zipcode and so forth.  Whereas, notification events probably can't
be sanitized until they've been "sent", and you want to aggregate
them before (or when) you "send" them.  Perhaps different types
of events need different policies for when to do these things?

Okay, I've probably rambled again, so I'll quit now.
For the moment.