[OPEN-ILS-GENERAL] ***SPAM*** Re: [OPEN-ILS-DEV] Sustainable QA

[steven]

I am a casual developer in the Evergreen community. Here are comments of mine
about some excerpts from Sustainable QA that I find particularly interesting.
The comments are primarily technical in nature but I think of general interest.


p.4, 'Providing example tests at each major level of the Evergreen “stack” and
making it straightforward to run such tests whenever a copy of the source code
is run through the build procedure will significantly facilitate the writing of
additional unit tests by any party.'

As a casual contributor to the code, I look forward to having the above state
of affairs. It's been about seven years since the first releases of the
software, and I do not understand why the testing framework is not more mature.
Whenever I contribute a bit of new logic to the Perl code, for example, I am
stymied by not being able to contribute a unit test to show evidence to others
that the new logic works.


p.4, 'Today Evergreen has a handful of unit tests in Perl.'

p.1, 'Some components of QA exist today in Evergreen and OpenSRF. We have Perl
and C unit test frameworks. We have a load test driver in Constrictor. We have
continuous integration infrastructure using Buildbot to run basic code
verification.'

The Buildbot[1] is currently not very capable and therefore not very significant
in its impact.  Of the 315 tests that the Buildbot reports as succeeding, most of
them check if Perl modules are installed, but do not execute any source code.
Of the few unit tests that actually runs, none of them can test an OpenSRF
service call nor perform a database operation.  This is because the 'continuous
integration infrastructure' takes effect during 'installation time' and does
not presume the OpenSRF framework or the Postgres database is installed and
running.


p.4, 'To effectively test the Evergreen server as a whole it is necessary to
issue API requests against a running server with sample data. Such requests
exercise each layer of Evergreen, including the database, the backend modules
and the code that allows them to communicate'

To achieve the above, the Buildbot must have OpenSRF (and ejabberd, the XMPP
server) installed.  If there was a way to 'stub out' OpenSRF, many functional
tests could be run earlier in the process, ie, without setting up a fully
running server.  By 'stubbing out' OpenSRF, I mean to have a situation where
the OpenILS code runs with OpenSRF replaced by something simpler[2].


[1] http://testing.evergreen-ils.org/buildbot/waterfall

[2] I have been investigating how OpenSRF can be stubbed out; it looks like
some systemic code changes are needed in the OpenILS application modules in
order for a simple stub to work.

-- 
steven chan
schan at sitka.bclibraries.ca