[OPEN-ILS-DEV] Evergreen and QA

[Dan Scott]

On Wed, Feb 13, 2013 at 02:38:22PM -0500, W. Brad LaJeunesse wrote:
> Fellow Evergreeners:
> 
> At the "future of the staff client" developer meeting Quality Assurance
> (QA) came up repeatedly. This is a serious issue that affects every one of
> us. It obviously affects all users of Evergreen, whether staff or customer,
> but it also greatly affects Equinox in that bugs and other software defects
> increase our support load and also decreases overall satisfaction with the
> Evergreen product. It's an issue we've been deeply concerned about for some
> time.
> 
> A few years ago, there was a formal and concerted QA effort. GPLS was
> generous to fund it, and we were making good headway with nightly builds,
> tests, etc. However, with funding cuts and other issues becoming priority,
> QA has largely been ignored and we've all felt the result. As a community,
> we need to correct this.

Hmm, if you're talking about the work that Shawn Boyette was doing
in 2009, didn't Equinox basically destroy all evidence of his
work--wiping the continuous integration server he had set up, removing
the code that he had developed to run the contiunous integration server,
etc, in February 2010? Sadly, I think GPLS' generous funds weren't put
to the best use. We do still have the Perl module build work that Shawn
put into place for OpenSRF, thankfully, and some unit tests in OpenSRF
as a legacy of his work.

After Shawn was let go, I adopted the buildbot.net continuous
integration framework (same name as what Shawn had been working on but
an existing open source framework with a community) in January 2011, and
a few months later set up a server on a VM that Equinox generously
supplied. I also set up Evergreen to support Perl unit testing,
following Shawn's lead with OpenSRF.
(http://coffeecode.net/archives/244-Evergreens-continuous-integration-servers-past,-present,-and-future.html)

That buildbot ran for about a year, then the VM died for a few months
waiting on Equinox to bring it back to life. Galen resurrected it, and I
got the buildbot running again in April 2012, and it ran until somewhere
around December 2012, when apparently the VM died and was not
recoverable.

Equinox set up another new VM about a week ago to offer to the community
for a testing server, and I've once again started setting up the
continuous integration environment. The server is now alive and running
unit tests on every commit, but I still need to proxy port 8010 to make
it easily visible on the web.

So your assessment that "QA has largely been ignored" might be true from
one perspective, but I've been making noise about it (see, for example,
http://coffeecode.net/archives/228-Got-funds-for-enhancing-Evergreen-Looking-for-places-to-spend-it.html).
and working on improving it for a long time (but also perhaps largely
being ignored; Others in the community have contributed unit tests;
Kevin Beswick, for example, added some unit tests for C libraries, and
way back when David Fiander wrote unit tests for his MHFD Perl modules.
GPLS and UPEI have donated build slaves to the testing effort. We also
had a pretty serious discussion about testing and QA efforts at the
Hack-a-way. James Fournie's recent suggestion to adopt Cucumber (or
something like it) is another sign of QA not-being-ignored by the
broader community.

> Equinox would like to be part of the solution and we very much want to
> re-launch and lead the QA effort, but we need the community's help and
> support. We have a formative plan and we have resources we can devote, but
> we need funding to make it happen.
>
> We believe that a 3-month period is enough to get started and see if this
> effort is fruitful. We're looking for $30,000 to fund a 3-month effort. If
> we're successful, then we'll search for further funding to keep rolling.

I'll be frank, as I was in IRC a few days ago: my concern with Equinox
leading the QA effort is that, as far as I can tell, Equinox doesn't
really have QA expertise to offer. Galen gave a pretty convincing
lightning talk on pgTAP, but we (unfortunately) haven't adopted it.
Bill built the constrictor framework for stress-testing and performance,
but it's not been integrated into the Evergreen processes in any way.
And in the end, you can have all the best tools in the world, but if you
don't use them and you jam code without tests into a release branch
because you want to meet a deadline, then your QA is still going to
suffer.

Rather than relying on Equinox to bootstrap themselves into a position
of understanding how to implement QA (including not just tools but
process changes and development culture changes--a tall order for 3
months), why don't we approach a company that has demonstrated expertise
with QA and see how far $30K would go towards getting them to help us
make the technical and cultural changes to achieve QA success?

I'd suggest OmniTI as one such possible company (disclosure: I know some
good people who have worked there over the years). I bet if we contacted
some people at Mozilla like Jonathan Nightingale he could point us at
XUL-savvy QA experts.