[OPEN-ILS-DEV] Release process & QA thoughts

[Dan Scott]

As the Evergreen community grows and more people are familiar /
comfortable with installing OpenSRF and Evergreen, I wonder whether we
could adopt a slightly more formal process for releases to avoid some
of the "brown paper bag" problems that turned up in the OpenSRF 1.0.0
and Evergreen 1.4.0.0-rc1 release. I should note in advance that I was
inspired by a presentation that Jonathan Nightingale of the Mozilla
Foundation gave at the Free Software Open Source Symposium last week
on security & QA processes that Firefox has adopted
(http://blog.johnath.com/2008/07/02/the-most-important-thing/). One of
his core points was: "The single most important thing you can do is
find ways to capture expensive knowledge so that you never pay for the
same lesson twice."

I would like to put forward a suggestion that no major release (x.y.0
in the case of OpenSRF or x.y.z.0 in the case of Evergreen) should
ever go out without at least one release candidate. OpenSRF 1.0.0 was
released without a release candidate to support Evergreen 1.4.0.0,
but:
  * it turned out to have a major bug that prevented CGI processes
from working properly on Apache
  * the optional Python support was broken
  * the README documented incorrect configure options
  * the sample opensrf.xml configuration file contained a few errors
that chewed up an hour or two of head-scratching time

A release candidate would have given members of the community the
chance to catch some or all of these problems before we put together a
final release. To be fair, most of these problems are masked in the
current typical use case where the Evergreen install instructions are
followed, no Python is used, and the Evergreen config file gets copied
over top of the OpenSRF config file.

I would also like to propose more open communication about upcoming
releases on the -dev list to help coordinate efforts. For the OpenSRF
1.0.0 / Evergreen 1.4.0.0-rc1 schedule, as the end of September
approached we had agreed on October 17th as a target date for a
release. However, during the final week we were largely unable to
communicate on IRC due to conflicting schedules (it looks like about a
dozen libraries migrated to Evergreen during that week, and I took off
on vacation on the 17th). If the core developers can't at least give a
round of "+1"s on the list to a release go/no-go proposal, it might be
better to delay the release until we can catch our breath rather than
rushing out a release to meet our rather arbitrary release date. I
left open the option of cutting an immediate "pure-bugfix" release in
the previous suggestion to cover the event that a severe security
vulnerability is discovered - although even then, the core devs should
have a chance to vet and test the required changes before releasing it
into the wild.

As a short term measure to help the community help us catch problems
in release candidates or releases, a release-testing checklist might
help out. For example, when a release is posted to the download site,
a thread should be started on the list (-dev probably) that announces
the release and asks for tester feedback. The initial post in the
thread could contain a copy of a tester's checklist that we maintain
on the wiki - as we discover new ways that we can break things, we can
add tests to the wiki. Responses to the post in the thread could then
contain a list of the tests that the particular tester ran, on which
platform, whether it was an upgrade or a clean install, etc; something
like:

* Summary: Mostly looks good; Python support in OpenSRF might be
broken and had problems with placing holds in the OPAC. Didn't try
SRU, Z39.50, or i18n.
* Platform: Ubuntu Hardy (64-bit)

OpenSRF details:
  * Version: 1.0.1
  * Clean install or upgrade: Upgrade from 1.0.0
  * Output of osrf_config: (paste output of 'osrf_config' command here)
  * Tests:
    * srfsh: math_bench: success
    * Python client: Failed with the following exception: <blah>
    * Perl client: success
    * gateway test: success
    * http_translator test: success
    * i18n tests: not attempted

Evergreen details:
  * Version: 1.4.0.1
  * Database: (output some automated output with DB version, libpq
version, locale settings, EG schema version)
  * Clean install or upgrade: Upgrade from 1.4.0.0-rc1
  * Output of eg_config: (paste output of 'eg_config' command here)
  * Tests:
    * srfsh: login admin open-ils: success
    * Web - browser = Firefox 3.0.1 on Windows Vista
      * vandelay importer: success (loaded test datasets 3, 4, and 5)
      * OPAC login: success
      * OPAC search: success
      * OPAC place hold: failed (request appeared to succeed, but list
of holds was not updated in "My Account")
      * Simple self-check: success
    * Staff client: xulrunner 1.9.1 package running on Ubuntu 64-bit Hardy
      * OPAC login: success
    * bulk importer: success
    * i18n tests: not attempted
    * SRU tests: not attempted
    * Z39.50 tests: not attempted

Maybe a set of release-testing volunteers could organize themselves
(via the list / wiki) and we could give volunteers a week to test,
report, and resolve problems with a release candidate before cutting a
final release?

In an ideal world, of course, enterprising testers would create unit
tests and system tests that help automate some or all of these tests
and report the data with much less effort and in a more consistent
way. Projects like PHP have a "make test" target that give the user
the option of automatically submitting the result of running all unit
tests to a QA mailing list; this helps PHP devs determine where a
common problem might exist (e.g. xulrunner on 64-bit Ubuntu). Also in
an ideal world, we would add new unit tests concurrent with any new
feature (Nightingale says that at Firefox "nothing lands without
tests... let me repeat that: Nothing. Lands. Without. Tests.") but
perhaps that ideal world has to remain ideal until someone with
regression testing expertise can assist us in setting up a framework
and seed tests (either in a volunteer capacity or in a hired-gun
capacity).

I wonder, too, whether the introduction of major new features (such as
i18n support) should require some level of documentation in the wiki
(description, configuration steps, tests to perform) and have at least
one other person reproduce success in following the configuration
steps and tests before making the feature generally available as part
of a release. It's not enough for me as the primary developer of a
feature to nod and say "Yep, that's ready" because I'm too close to
the feature to notice when I'm skipping instructions or assuming
something that needs to be explicit - I need someone else to confirm
that it's ready so that the rest of the community has a chance to
succeed in trying it out. For 1.4.0.0-rc1, for example, Mike wasn't
able to follow my instructions at
http://open-ils.org/dokuwiki/doku.php?id=evergreen-admin:customizations:i18n
to generate a release tarball that included the translations for fr-CA
and hy-AM - meaning that I had not done a good enough job of
documenting the required steps for the release coordinator. Combined
with the focus on meeting our somewhat arbitrary Oct. 17th deadline,
it meant that the first release candidate of the "internationalization
release" went out without any significant examples of the
internationalization support we had been touting for some time. Note
that this wiki documentation could then be linked to from the "Release
features" page for people that want more information (both descriptive
and prescriptive) - and could also be used as raw materials by doc
writers.

I hope that this set of suggestions is taken in the spirit in which it
is offered. One of the challenges that I predicted for this project
back in early 2007 was the difficulty in growing a small project into
a larger project. Evergreen has enough functionality now that it's
probably beyond the capabilities of even the best release-wrangler
(and in Mike we definitely have one of the best) to single-handedly
vet every corner of the code before cutting a release. I believe that
these suggestions will help the community help the Evergreen project
on the QA side, and that that in turn will enable the developers to
spend more time on development and less time trying to help people
debug problems with our releases.

-- 
Dan Scott
Laurentian University