[OPEN-ILS-DEV] Release process & QA thoughts

[Karen Schneider]

I'd like to provide a high-level summary of what I think Dan said, as he's
hitting on at least three very important issues. Basically I see him arguing
quite eloquently for better...

1. Communication: "If the core developers can't at least give a round of
"+1"s on the list to a release go/no-go proposal, it might be better to
delay the release until we can catch our breath rather than
rushing out a release to meet our ... release date." Makes sense.

2. Documentation: It needs to be integrated into the product development
timeline. Yes. Dan not only argues for the documentation but provides a path
for suggesting minimum requirements for release documentation. I would even
go so far as to say if at least the modicum of documentation Dan proposes is
not ready, then per #1, "it might be better to delay the release."

3. Testing/QA: Absolutely. Dan argues for both the need and for the
methodology (checklist, confirmed testers, timeline). I'm tempted to paste
his draft checklist into a wiki document. As with #1 and #2, this sounds
like a logical part of the release process.


-- 
| Karen G. Schneider
| Community Librarian
| Equinox Software Inc. "The Evergreen Experts"
| Toll-free: 1.877.Open.ILS (1.877.673.6457) x712
| E-Mail/AIM: kgs at esilibrary.com
| Web: http://www.esilibrary.com


On Sun, Oct 26, 2008 at 4:34 PM, Dan Scott <denials at gmail.com> wrote:

> As the Evergreen community grows and more people are familiar /
> comfortable with installing OpenSRF and Evergreen, I wonder whether we
> could adopt a slightly more formal process for releases to avoid some
> of the "brown paper bag" problems that turned up in the OpenSRF 1.0.0
> and Evergreen 1.4.0.0-rc1 release. I should note in advance that I was
> inspired by a presentation that Jonathan Nightingale of the Mozilla
> Foundation gave at the Free Software Open Source Symposium last week
> on security & QA processes that Firefox has adopted
> (http://blog.johnath.com/2008/07/02/the-most-important-thing/). One of
> his core points was: "The single most important thing you can do is
> find ways to capture expensive knowledge so that you never pay for the
> same lesson twice."
>
> I would like to put forward a suggestion that no major release (x.y.0
> in the case of OpenSRF or x.y.z.0 in the case of Evergreen) should
> ever go out without at least one release candidate. OpenSRF 1.0.0 was
> released without a release candidate to support Evergreen 1.4.0.0,
> but:
>  * it turned out to have a major bug that prevented CGI processes
> from working properly on Apache
>  * the optional Python support was broken
>  * the README documented incorrect configure options
>  * the sample opensrf.xml configuration file contained a few errors
> that chewed up an hour or two of head-scratching time
>
> A release candidate would have given members of the community the
> chance to catch some or all of these problems before we put together a
> final release. To be fair, most of these problems are masked in the
> current typical use case where the Evergreen install instructions are
> followed, no Python is used, and the Evergreen config file gets copied
> over top of the OpenSRF config file.
>
> I would also like to propose more open communication about upcoming
> releases on the -dev list to help coordinate efforts. For the OpenSRF
> 1.0.0 / Evergreen 1.4.0.0-rc1 schedule, as the end of September
> approached we had agreed on October 17th as a target date for a
> release. However, during the final week we were largely unable to
> communicate on IRC due to conflicting schedules (it looks like about a
> dozen libraries migrated to Evergreen during that week, and I took off
> on vacation on the 17th). If the core developers can't at least give a
> round of "+1"s on the list to a release go/no-go proposal, it might be
> better to delay the release until we can catch our breath rather than
> rushing out a release to meet our rather arbitrary release date. I
> left open the option of cutting an immediate "pure-bugfix" release in
> the previous suggestion to cover the event that a severe security
> vulnerability is discovered - although even then, the core devs should
> have a chance to vet and test the required changes before releasing it
> into the wild.
>
> As a short term measure to help the community help us catch problems
> in release candidates or releases, a release-testing checklist might
> help out. For example, when a release is posted to the download site,
> a thread should be started on the list (-dev probably) that announces
> the release and asks for tester feedback. The initial post in the
> thread could contain a copy of a tester's checklist that we maintain
> on the wiki - as we discover new ways that we can break things, we can
> add tests to the wiki. Responses to the post in the thread could then
> contain a list of the tests that the particular tester ran, on which
> platform, whether it was an upgrade or a clean install, etc; something
> like:
>
> * Summary: Mostly looks good; Python support in OpenSRF might be
> broken and had problems with placing holds in the OPAC. Didn't try
> SRU, Z39.50, or i18n.
> * Platform: Ubuntu Hardy (64-bit)
>
> OpenSRF details:
>  * Version: 1.0.1
>  * Clean install or upgrade: Upgrade from 1.0.0
>  * Output of osrf_config: (paste output of 'osrf_config' command here)
>  * Tests:
>    * srfsh: math_bench: success
>    * Python client: Failed with the following exception: <blah>
>    * Perl client: success
>    * gateway test: success
>    * http_translator test: success
>    * i18n tests: not attempted
>
> Evergreen details:
>  * Version: 1.4.0.1
>  * Database: (output some automated output with DB version, libpq
> version, locale settings, EG schema version)
>  * Clean install or upgrade: Upgrade from 1.4.0.0-rc1
>  * Output of eg_config: (paste output of 'eg_config' command here)
>  * Tests:
>    * srfsh: login admin open-ils: success
>    * Web - browser = Firefox 3.0.1 on Windows Vista
>      * vandelay importer: success (loaded test datasets 3, 4, and 5)
>      * OPAC login: success
>      * OPAC search: success
>      * OPAC place hold: failed (request appeared to succeed, but list
> of holds was not updated in "My Account")
>      * Simple self-check: success
>    * Staff client: xulrunner 1.9.1 package running on Ubuntu 64-bit Hardy
>      * OPAC login: success
>    * bulk importer: success
>    * i18n tests: not attempted
>    * SRU tests: not attempted
>    * Z39.50 tests: not attempted
>
> Maybe a set of release-testing volunteers could organize themselves
> (via the list / wiki) and we could give volunteers a week to test,
> report, and resolve problems with a release candidate before cutting a
> final release?
>
> In an ideal world, of course, enterprising testers would create unit
> tests and system tests that help automate some or all of these tests
> and report the data with much less effort and in a more consistent
> way. Projects like PHP have a "make test" target that give the user
> the option of automatically submitting the result of running all unit
> tests to a QA mailing list; this helps PHP devs determine where a
> common problem might exist (e.g. xulrunner on 64-bit Ubuntu). Also in
> an ideal world, we would add new unit tests concurrent with any new
> feature (Nightingale says that at Firefox "nothing lands without
> tests... let me repeat that: Nothing. Lands. Without. Tests.") but
> perhaps that ideal world has to remain ideal until someone with
> regression testing expertise can assist us in setting up a framework
> and seed tests (either in a volunteer capacity or in a hired-gun
> capacity).
>
> I wonder, too, whether the introduction of major new features (such as
> i18n support) should require some level of documentation in the wiki
> (description, configuration steps, tests to perform) and have at least
> one other person reproduce success in following the configuration
> steps and tests before making the feature generally available as part
> of a release. It's not enough for me as the primary developer of a
> feature to nod and say "Yep, that's ready" because I'm too close to
> the feature to notice when I'm skipping instructions or assuming
> something that needs to be explicit - I need someone else to confirm
> that it's ready so that the rest of the community has a chance to
> succeed in trying it out. For 1.4.0.0-rc1, for example, Mike wasn't
> able to follow my instructions at
>
> http://open-ils.org/dokuwiki/doku.php?id=evergreen-admin:customizations:i18n
> to generate a release tarball that included the translations for fr-CA
> and hy-AM - meaning that I had not done a good enough job of
> documenting the required steps for the release coordinator. Combined
> with the focus on meeting our somewhat arbitrary Oct. 17th deadline,
> it meant that the first release candidate of the "internationalization
> release" went out without any significant examples of the
> internationalization support we had been touting for some time. Note
> that this wiki documentation could then be linked to from the "Release
> features" page for people that want more information (both descriptive
> and prescriptive) - and could also be used as raw materials by doc
> writers.
>
> I hope that this set of suggestions is taken in the spirit in which it
> is offered. One of the challenges that I predicted for this project
> back in early 2007 was the difficulty in growing a small project into
> a larger project. Evergreen has enough functionality now that it's
> probably beyond the capabilities of even the best release-wrangler
> (and in Mike we definitely have one of the best) to single-handedly
> vet every corner of the code before cutting a release. I believe that
> these suggestions will help the community help the Evergreen project
> on the QA side, and that that in turn will enable the developers to
> spend more time on development and less time trying to help people
> debug problems with our releases.
>
> --
> Dan Scott
> Laurentian University
>



--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20081027/8c38d3f8/attachment-0001.htm