[OPEN-ILS-DEV] Proposal: Versioned Add-on Packages

[Mike Rylander]

In order to make extending Evergreen data structures as straight
forward as possible, we use an abstraction layer that has at its heart
an XML file called, by default, fm_IDL.xml.  This file describes the
structure of and relationship between objects that Evergreen's
component applications have access to, and tells the storage subsystem
how to get to the underlying data stored in Postgres.  All database
interaction is driven and directed by this file, and there are
explicit provisions built into this abstraction layer for the purpose
of extending the stock database schema with custom tables and views.
Local customizations can also be made to existing objects, extending
them in ways that allow local persistent fields to be added, and new
relationships to be described, all without changing any existing code
that uses these objects (with the exception of removing existing,
in-use fields -- that would be bad, m'kay?).

However, while the current infrastructure is very flexible in terms of
both core development and local extension, upgrades can be less then
simple.  The new IDL needs to be patched with any local
customizations, and particular care needs to be taken when new core
features supersede existing local extensions.  Also, this only
addresses the description of the database.  No versioning of the local
schema extensions takes place, nor is any local client code version
controlled, and could be rendered nonfunctional by an upgrade.

To help address these issues, I propose we institute a versioning and
installation convention and process for adding extensions to
Evergreen, so that upgrades can be handled in a less chaotic manner
and we can help extension developers (of which I am, of course, one)
avoid some pain.

First, we need a mechanism for extending fm_IDL.xml in such a way that
upgrades will not break locally added table and view descriptions.
For this, I propose a new standard directory which will live under
/openils/conf (or the configured, er, config dir) which will hold
extension XML files.  These files will contain new IDL class
definitions.  Imagine, for example, a file called
money.open_balance_by_usr_home_and_owning_lib.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<class
  id="rmobbhol"
  version="rmobbhol-2009.02.19"
  controller="open-ils.reporter-store"
  oils_obj:fieldmapper="reporter::money::open_balance_by_usr_home_and_owning_lib"
  oils_persist:tablename="money.open_balance_by_usr_home_and_owning_lib"
  reporter:core="true"
  reporter:label="Open Circulation Balance by User Home Library and
Owning Library">
    <fields oils_persist:primary="home_ou">
        <field name="isnew" oils_obj:array_position="0"
oils_persist:virtual="true" />
        <field name="ischanged" oils_obj:array_position="1"
oils_persist:virtual="true" />
        <field name="isdeleted" oils_obj:array_position="2"
oils_persist:virtual="true" />
        <field reporter:label="User Home Library" name="home_ou"
oils_obj:array_position="3" oils_persist:virtual="false"
reporter:datatype="org_unit"/>
        <field reporter:label="Owning Library" name="owning_lib"
oils_obj:array_position="4" oils_persist:virtual="false"
reporter:datatype="org_unit"/>
        <field reporter:label="Billing Types" name="billing_types"
oils_obj:array_position="5" oils_persist:virtual="false"
reporter:datatype="text"/>
        <field reporter:label="Balance" name="balance"
oils_obj:array_position="6" oils_persist:virtual="false"
reporter:datatype="money"/>
    </fields>
    <links>
        <link field="owning_lib" reltype="has_a" key="id" map="" class="aou"/>
        <link field="home_ou" reltype="has_a" key="id" map="" class="aou"/>
    </links>
</class>

(NOTE: This particular view definition already exists in the stock IDL
as an example extension, though without the (newly invented) version
attribute.)

In order to integrate the content of these files into the main IDL, I
propose that we change the name of the stock IDL file to
fm_IDL_base.xml.  Then, we extend autogen.sh such that it takes the
content of each file in the extension directory and adds it under the
root node of fm_IDL_base.xml (taking care to avoid replacing entities
with their literal counterparts), generating an entirely new file
which will be placed at fm_IDL.xml.

In addition to installing these extensions to the IDL, a versioned
add-on should also take advantage of the config.upgrade_log table,
inserting information about what it is installing within its schema
creation script.  This is more about schema future-proofing than
anything else.  It will allow us to avoid using table names that are
in use by known extensions, and it will allow extension upgrade
scripts to know the state of the extension schema, if that is
important.  Something like:

-------8<-----------

BEGIN;

INSERT INTO config.upgrade_log (version) VALUES ('rmobbhol-2009.02.19');

CREATE OR REPLACE VIEW money.open_circ_balance_by_usr_home_and_owning_lib AS
    SELECT  circ.id,
        usr.home_ou,
        cn.owning_lib,
        bill.billing_type,
        SUM(bill.amount) AS billed
      FROM  action.circulation circ
        JOIN money.billing bill ON (circ.id = bill.xact)
        JOIN asset.copy cp ON (circ.target_copy = cp.id)
        JOIN asset.call_number cn ON (cn.id = cp.call_number)
        JOIN actor.usr usr ON (circ.usr = usr.id)
      WHERE circ.xact_finish IS NULL
        AND NOT bill.voided
      GROUP BY 1,2,3,4
      ORDER BY 1,2,3,4;

COMMIT;

-------8<-----------

Installation of an add-on should be handled by a script that does at
least the following:
 * It should use the eg_config utility (1.4.0.0 and beyond) with the
--version switch to decide if the Evergreen version will support the
add-on, and possibly how to install it.
 * If any external dependencies exist, it should test for their
existence and refuse to run until they are fulfilled
 * If a new table or view is to be installed directly into the
database (in 1.4 and beyond, virtual views can be constructed in the
IDL alone with no need to touch the db), then the script should ask
the settings server for the appropriate database connection
information.  It is not uncommon to have reporting-oriented tables and
views installed only on a dedicated reporting server, and not on any
of the front-end database instances.
 * The script should then check the config.upgrade_log table to make
sure that the schema it wants to put in place is not already there.
If it is, then the script should skip schema setup
 * If an older version of the schema is there, the script should be
prepared to run one or more upgrade scripts, and add the new version
information to config.upgrade_log.
 * It should use the eg_config utility with the --sysconfdir switch to
find out where to place the IDL addition file(s) it needs.
 * It should likewise use eg_config utility with the --bindir switch
to find out where to place new server-side binaries or scripts.


A few more points of thought:
 * Add-ons should attempt to use a very specific prefix for their
oils_obj:fieldmapper attribute in the IDL addition.  The example above
does /not/ do this ...
 * We should invent a convention for prefixing the id attribute, in
particular, so that there is little chance of collision.  Perhaps
"addon_"?
 * We should set aside an add_on schema for add-ons to use, so that
the stock database layout does not get "polluted" with non-core tables
and views.

This proposal does not cover:
 * Interface (Staff Client and OPAC) changes.  That can come later and
is a whole different ball of wax, but should eventually extend this
proposal.

Finally, I propose that all of this should be stored and managed in
the ILS-Contrib subversion repository.  Specific versions should be
svn-copy'd to a tag directory, and release tarballs should be built
from these "tagged" releases in order to provide a specific, know-good
version for distribution, which may be specific to Evergreen releases
depending on what they require of the stock system.

I am currently working on a project that would both benefit from this
sort of tracking and serve as a pretty good example of just how to go
about building an add-on package.  If this proposal is accepted by the
general community then I will begin working on an add-on package for
the 1.2 and 1.4.0 series that will bootstrap this infrastructure.
Install that and you will get the backend stuff (an add_on schema, an
updated autogen.sh that knows how to construct fm_IDL.xml) needed to
install more add-ons.  I will also work on the IDL-mangling bits in
trunk so the next major release will be able to natively handle IDL
additions, as well as work on an add-on to teach autogen how to do
this in 1.2 and 1.4.

Comments/suggestions/additions encouraged, and thanks in advance!

-- 
Mike Rylander
 | VP, Research and Design
 | Equinox Software, Inc. / The Evergreen Experts
 | phone:  1-877-OPEN-ILS (673-6457)
 | email:  miker at esilibrary.com
 | web:  http://www.esilibrary.com