[OPEN-ILS-DEV] Notification system design

[Mike Rylander]

It seems we may have meandered off the path a bit into a field of
specifics ... I'll take some blame for that by introducing examples
without qualifying them properly.  I wanted to use them just to
illustrate the difference between (or, perhaps, lack thereof) some
common situations, but I think it's too early to worry about specific
cases (assuming we design this thing correctly).

>From the thoughts so far, it feels like we are all thinking of a
similar subsystem.  Looking at it from the broadest possible view,
outside an ILS perspective, here's what I see:

* a queuing system that can accept an event, some related data, and a
time (relative or absolute, "now" or "later") to fire the event, with
ASAP or long-delay capability (say, within 5 minutes or now + 1
month).

* a generalized event creation framework, usable by other subsystems
and potentially exposed via some interface for users to create events
on demand

* an interface for dequeuing events that needn't run.  perhaps part of
this could be built into the event creation subsystem -- you tell the
event creation interface about the conditions required to fire the
event and if those conditions no longer exist at run time then the
event is discarded.

* a generalized event aggregation system, such that like events can be
grouped together (potentially forming one or more "master" events, or
maybe just selecting grouped events based on aggregation criteria) for
batch processing where it makes sense

* a generalized event processing framework which will take an event
(or events, or a master event) and hand it to the proper logic module
(read: plugin).  these logic modules may have side effects (billing,
emailing, creating another event, etc)

The first and second there are fairly simple.  It's just a matter
creating a table and defining an interface.  Even storing auxiliary
data is easy ... in the simplest case, just accept an object (shaped
like a hash) and store a serialized version on the event table which
then gets deserialized and passed into the processing logic at event
run time.  We have mechanisms that do this for caching expensive (to
create) objects.

Event aggregation is a bit more tricky.  It isn't technically hard,
but it's also not easy to make it flexible when there's over 100
object types, most linked in some way, all with useful data hidden
behind those links.  If we were building _just_ a hold event queue,
then it's easy, but if we want it to be generalized for patron events,
and circulation events, and payment events and who knows what else,
then it's less simple.  This could potentially be simplified at the DB
level by judicious use of views (among other techniques), so it may
not be as bad as I fear.

Depending on the requirements of the event, the processing logic may
be simple and just fill in templates.  I think nearly any email
notification for any purpose (including aggregated notices via email)
could be handled by such a thing without any real specialization
outside the template proper.

Anyway, just wanted to get some blue-sky thoughts out there.  Is this
generally what you guys have been seeing in your heads?  (It's an
extension and generalization of what we've discussed here at PINES in
the past, JFYI.)


I've addressed several points below from Nathan's email because I
think it may be useful for everyone to see what is and isn't already
built, and in what way.  (So, read on... ;) )


On 12/28/06, Nathan Eady <eady at galion.lib.oh.us> wrote:
> Mike Rylander wrote:
> > There are two competing types of "triggers" we'll need to consider
> > here:  real-time, action based triggers (hold arrives at its
> > destination; fine is generated on an overdue item), and time span
> > expiration triggers (item is 14 days overdue; hold fulfillment is
> > taking forever, would you like to try other formats/editions?).
>
> There are also the "queue it for the next time we do this" kind,
> which is what would be needed e.g. for old fashioned postal
> notifications and billers.  This gets a little complicated because
> if the item is added to a queue by a trigger event, then later when
> the queue is processed something has to check whether it's still
> appropriate, and also whether it can be combined with other
> notices going to the same address, so the queue may need fields
> it otherwise wouldn't require (e.g., patron record ID, item
> record ID) to allow for this sort of checking.
>

The other option for /not/ running a queued event is to have a
dequeuing mechanism.  Something like {when an item is returned,
disable all events of type X that point to object Y which is of class
Z}.

The main thing you need there is some expiration threshold.  As for
event consolidation, that sounds like suspiciously like a job for a
processing time aggregation algebra to me.  ;)  This is something I've
had success with in the past -- it's roughly equivalent to event
aggregation logic in network/system monitoring software, and as long
as we don't get too crazy with the breadth of allowable conditions it
can have a decent/simple UI.

> And yes, postal notification is going to be worth implementing
> at some point.  In small, somewhat technophobic Galion, we
> currently have email addresses for roughly half of one percent
> of our registered patrons.  That number will go up over time,
> but I think this illustrates that postal notification is not
> going to be a thing of the past -- at least, not for all
> libraries -- for a good number of years, perhaps decades.

Oh, sure.  No argument there.  There are currently scripts that
generate data for postal notification, including aggregating several
types of notifications into one block.  They spit out a simple XML
file and are controlled by cron.  Bill can get into more of that when
he returns... :)

>
> Also, some large library systems want to handle notifications,
> especially postal ones, in a decentralized fashion, with each
> library or branch mailing out its own (so that the postage
> comes out of the correct budget, the correct letterhead is
> used, or for whatever reason).  So you end up having to decide

[snip]

Sure ... Those sorts of policy settings can be handled by the
Organizational Unit Settings infrastructure.  That gives us an
extensible framework for creating and applying settings at different
levels of the organizational hierarchy.  There is an analogous User
Settings, um, setup. :)

> It seems reasonable that "pending" notifications (i.e., those that
> have been triggered or queued but not yet sent) should inhabit a
> table in the database.

Oh, of course.  I meant in terms of historical reporting.  I hesitate
to through away any data if I can avoid it, but even notifications are
personally identifiable data...

>
> A further complication is that some libraries expect to send
> multiple notices at different intervals -- e.g., a library might
> want to send a "warning" notice shortly before the grace period
> expires, an "overdue" notice some number of days after the grace
> period expires, a "long overdue" notification a month or so
> later, and a "biller" when the item is charged to the patron's
> account.  Each of these might have somewhat different wording
> and possibly contain different details about the item, and in
> a some cases you might be sending more than one level of these
> notifications to the same patron at the same time.
>

That's currently triggered via external scripts.  And, again, this is
relatively simple business logic -- much more so than, say, item hold
eligibility.

Templates will need to be part of the overall system, perhaps
themselves contained in OU Settings.  Making templates smart enough to
fetch bits related to the event seed should be do-able ... or, it's
one option.


> place.  You probably want to get the checkin staff's attention
> anyhow, so they can place the item on the hold shelf instead of
> the reshelving cart (or whatever), so it might be reasonable to
> ask the user something along the lines of "There is a hold
> request on this item.  Hold it on the reserve shelf?" with
> options like "Hold Now", "Later", and maybe "Cancel Hold".

I think I caused some confusion with my example.  I only meant to give
an example of a "not this very second, but RSN" type event.  The staff
client and middle layer actually already handle all the hold logic --
I was only interested in the short notification delay idea.

> Which brings up a further wrinkle:  ideally you'd want individual
> patron records to be marked "Don't use this type of notification",
> for any given type of notification that the library uses.  Then
> there needs to be a notification type priority list or schedule or
> whatever that can designate policies like, "For hold notifications,
> send email if possible, but if the patron has email notification
> disabled fall back to automated phonecall, and if that's disabled
> also then send a postcard."
>

That exists today.  The patron can set their hold notification
preferences in the My Account section, and the current email
notification code honors that flag.  If they have selected phone
notification then the staff is presented with the patron's phone
number on the hold shelf browser.  AFAICT, all of that can be extended
(and expanded) directly to other types of notification.  Yay for
reuse! :)

-- 
Mike Rylander
mrylander at gmail.com
GPLS -- PINES Development
Database Developer
http://open-ils.org