[OPEN-ILS-DEV] Proposal: Versioned Add-on Packages

[Duimovich, George]

Mike, 

I am very enthusiastic about any moves to lay down more infrastructure to support plug-in and local extensions architecture, especially anything that eases development and maintenance, etc. I had just put in a conference proposal that partly sought to elicit this kind of discussion from the dev crew (e.g. what's next, etc.) so this was a good way to start the day. Your proposal is very significant and timely IMHO.

This is another move closer to the dream scenario and why many of us are so enthusiastic about Evergreen: all the usual off-the-shelf 'turn-key' ILS functionality (already there or in development, etc.), but with excellent support to exploit and extend the underlying platform for what it is and what it can offer. 

Thanks for that update!

George Duimovich
NRCan Library / Bibliothèque RNCan
(613) 996-2101



-----Original Message-----
From: open-ils-dev-bounces at list.georgialibraries.org [mailto:open-ils-dev-bounces at list.georgialibraries.org] On Behalf Of Mike Rylander
Sent: February 20, 2009 12:26 AM
To: Evergreen Development Discussion List
Subject: [OPEN-ILS-DEV] Proposal: Versioned Add-on Packages

In order to make extending Evergreen data structures as straight forward as possible, we use an abstraction layer that has at its heart an XML file called, by default, fm_IDL.xml.  This file describes the structure of and relationship between objects that Evergreen's component applications have access to, and tells the storage subsystem how to get to the underlying data stored in Postgres.  All database interaction is driven and directed by this file, and there are explicit provisions built into this abstraction layer for the purpose of extending the stock database schema with custom tables and views.
Local customizations can also be made to existing objects, extending them in ways that allow local persistent fields to be added, and new relationships to be described, all without changing any existing code that uses these objects (with the exception of removing existing, in-use fields -- that would be bad, m'kay?).

However, while the current infrastructure is very flexible in terms of both core development and local extension, upgrades can be less then simple.  The new IDL needs to be patched with any local customizations, and particular care needs to be taken when new core features supersede existing local extensions.  Also, this only addresses the description of the database.  No versioning of the local schema extensions takes place, nor is any local client code version controlled, and could be rendered nonfunctional by an upgrade.

To help address these issues, I propose we institute a versioning and installation convention and process for adding extensions to Evergreen, so that upgrades can be handled in a less chaotic manner and we can help extension developers (of which I am, of course, one) avoid some pain.

First, we need a mechanism for extending fm_IDL.xml in such a way that upgrades will not break locally added table and view descriptions.
For this, I propose a new standard directory which will live under /openils/conf (or the configured, er, config dir) which will hold extension XML files.  These files will contain new IDL class definitions.  Imagine, for example, a file called
money.open_balance_by_usr_home_and_owning_lib.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<class
  id="rmobbhol"
  version="rmobbhol-2009.02.19"
  controller="open-ils.reporter-store"
  oils_obj:fieldmapper="reporter::money::open_balance_by_usr_home_and_owning_lib"
  oils_persist:tablename="money.open_balance_by_usr_home_and_owning_lib"
  reporter:core="true"
  reporter:label="Open Circulation Balance by User Home Library and Owning Library">
    <fields oils_persist:primary="home_ou">
        <field name="isnew" oils_obj:array_position="0"
oils_persist:virtual="true" />
        <field name="ischanged" oils_obj:array_position="1"
oils_persist:virtual="true" />
        <field name="isdeleted" oils_obj:array_position="2"
oils_persist:virtual="true" />
        <field reporter:label="User Home Library" name="home_ou"
oils_obj:array_position="3" oils_persist:virtual="false"
reporter:datatype="org_unit"/>
        <field reporter:label="Owning Library" name="owning_lib"
oils_obj:array_position="4" oils_persist:virtual="false"
reporter:datatype="org_unit"/>
        <field reporter:label="Billing Types" name="billing_types"
oils_obj:array_position="5" oils_persist:virtual="false"
reporter:datatype="text"/>
        <field reporter:label="Balance" name="balance"
oils_obj:array_position="6" oils_persist:virtual="false"
reporter:datatype="money"/>
    </fields>
    <links>
        <link field="owning_lib" reltype="has_a" key="id" map="" class="aou"/>
        <link field="home_ou" reltype="has_a" key="id" map="" class="aou"/>
    </links>
</class>

(NOTE: This particular view definition already exists in the stock IDL as an example extension, though without the (newly invented) version
attribute.)

In order to integrate the content of these files into the main IDL, I propose that we change the name of the stock IDL file to fm_IDL_base.xml.  Then, we extend autogen.sh such that it takes the content of each file in the extension directory and adds it under the root node of fm_IDL_base.xml (taking care to avoid replacing entities with their literal counterparts), generating an entirely new file which will be placed at fm_IDL.xml.

In addition to installing these extensions to the IDL, a versioned add-on should also take advantage of the config.upgrade_log table, inserting information about what it is installing within its schema creation script.  This is more about schema future-proofing than anything else.  It will allow us to avoid using table names that are in use by known extensions, and it will allow extension upgrade scripts to know the state of the extension schema, if that is important.  Something like:

-------8<-----------

BEGIN;

INSERT INTO config.upgrade_log (version) VALUES ('rmobbhol-2009.02.19');

CREATE OR REPLACE VIEW money.open_circ_balance_by_usr_home_and_owning_lib AS
    SELECT  circ.id,
        usr.home_ou,
        cn.owning_lib,
        bill.billing_type,
        SUM(bill.amount) AS billed
      FROM  action.circulation circ
        JOIN money.billing bill ON (circ.id = bill.xact)
        JOIN asset.copy cp ON (circ.target_copy = cp.id)
        JOIN asset.call_number cn ON (cn.id = cp.call_number)
        JOIN actor.usr usr ON (circ.usr = usr.id)
      WHERE circ.xact_finish IS NULL
        AND NOT bill.voided
      GROUP BY 1,2,3,4
      ORDER BY 1,2,3,4;

COMMIT;

-------8<-----------

Installation of an add-on should be handled by a script that does at least the following:
 * It should use the eg_config utility (1.4.0.0 and beyond) with the --version switch to decide if the Evergreen version will support the add-on, and possibly how to install it.
 * If any external dependencies exist, it should test for their existence and refuse to run until they are fulfilled
 * If a new table or view is to be installed directly into the database (in 1.4 and beyond, virtual views can be constructed in the IDL alone with no need to touch the db), then the script should ask the settings server for the appropriate database connection information.  It is not uncommon to have reporting-oriented tables and views installed only on a dedicated reporting server, and not on any of the front-end database instances.
 * The script should then check the config.upgrade_log table to make sure that the schema it wants to put in place is not already there.
If it is, then the script should skip schema setup
 * If an older version of the schema is there, the script should be prepared to run one or more upgrade scripts, and add the new version information to config.upgrade_log.
 * It should use the eg_config utility with the --sysconfdir switch to find out where to place the IDL addition file(s) it needs.
 * It should likewise use eg_config utility with the --bindir switch to find out where to place new server-side binaries or scripts.


A few more points of thought:
 * Add-ons should attempt to use a very specific prefix for their oils_obj:fieldmapper attribute in the IDL addition.  The example above does /not/ do this ...
 * We should invent a convention for prefixing the id attribute, in particular, so that there is little chance of collision.  Perhaps "addon_"?
 * We should set aside an add_on schema for add-ons to use, so that the stock database layout does not get "polluted" with non-core tables and views.

This proposal does not cover:
 * Interface (Staff Client and OPAC) changes.  That can come later and is a whole different ball of wax, but should eventually extend this proposal.

Finally, I propose that all of this should be stored and managed in the ILS-Contrib subversion repository.  Specific versions should be svn-copy'd to a tag directory, and release tarballs should be built from these "tagged" releases in order to provide a specific, know-good version for distribution, which may be specific to Evergreen releases depending on what they require of the stock system.

I am currently working on a project that would both benefit from this sort of tracking and serve as a pretty good example of just how to go about building an add-on package.  If this proposal is accepted by the general community then I will begin working on an add-on package for the 1.2 and 1.4.0 series that will bootstrap this infrastructure.
Install that and you will get the backend stuff (an add_on schema, an updated autogen.sh that knows how to construct fm_IDL.xml) needed to install more add-ons.  I will also work on the IDL-mangling bits in trunk so the next major release will be able to natively handle IDL additions, as well as work on an add-on to teach autogen how to do this in 1.2 and 1.4.

Comments/suggestions/additions encouraged, and thanks in advance!

--
Mike Rylander
 | VP, Research and Design
 | Equinox Software, Inc. / The Evergreen Experts  | phone:  1-877-OPEN-ILS (673-6457)  | email:  miker at esilibrary.com  | web:  http://www.esilibrary.com