[OPEN-ILS-DEV] ***SPAM*** Re: ***SPAM*** Re: Sustainable QA

[Joe Atzberger]

IMHO, at this point, after however many years, we should probably concede
that OpenSRF is really a part of Evergreen and unify the repositories (or
at least incorporate via submodule), with the specific benefits of ease of
install, enabling this kind of automated testability, etc.  This is (far)
less work than creating and maintaining OpenSRFstub.

As I see it, the main problem is fitting all the processes that Evergreen
uses under a given test aparatus.  At the very least, we can optionally
enable a series of tests against a known-good reference backend server
setup specifically for this purpose, allowing the tests to proceed
conventionally.  But the ideal (everything forked locally, processes
talking to each other) should at least be attempted.

I don't think the (daemon/multiprocess) limitations are entrenched in
BuildBot, but if they are then it we should consider replacing it.

--joe


On Wed, Apr 24, 2013 at 4:41 PM, steven <schan at vcn.bc.ca> wrote:

> I am a casual developer in the Evergreen community. Here are comments of
> mine
> about some excerpts from Sustainable QA that I find particularly
> interesting.
> The comments are primarily technical in nature but I think of general
> interest.
>
>
> p.4, 'Providing example tests at each major level of the Evergreen “stack”
> and
> making it straightforward to run such tests whenever a copy of the source
> code
> is run through the build procedure will significantly facilitate the
> writing of
> additional unit tests by any party.'
>
> As a casual contributor to the code, I look forward to having the above
> state
> of affairs. It's been about seven years since the first releases of the
> software, and I do not understand why the testing framework is not more
> mature.
> Whenever I contribute a bit of new logic to the Perl code, for example, I
> am
> stymied by not being able to contribute a unit test to show evidence to
> others
> that the new logic works.
>
>
> p.4, 'Today Evergreen has a handful of unit tests in Perl.'
>
> p.1, 'Some components of QA exist today in Evergreen and OpenSRF. We have
> Perl
> and C unit test frameworks. We have a load test driver in Constrictor. We
> have
> continuous integration infrastructure using Buildbot to run basic code
> verification.'
>
> The Buildbot[1] is currently not very capable and therefore not very
> significant
> in its impact.  Of the 315 tests that the Buildbot reports as succeeding,
> most of
> them check if Perl modules are installed, but do not execute any source
> code.
> Of the few unit tests that actually runs, none of them can test an OpenSRF
> service call nor perform a database operation.  This is because the
> 'continuous
> integration infrastructure' takes effect during 'installation time' and
> does
> not presume the OpenSRF framework or the Postgres database is installed and
> running.
>
>
> p.4, 'To effectively test the Evergreen server as a whole it is necessary
> to
> issue API requests against a running server with sample data. Such requests
> exercise each layer of Evergreen, including the database, the backend
> modules
> and the code that allows them to communicate'
>
> To achieve the above, the Buildbot must have OpenSRF (and ejabberd, the
> XMPP
> server) installed.  If there was a way to 'stub out' OpenSRF, many
> functional
> tests could be run earlier in the process, ie, without setting up a fully
> running server.  By 'stubbing out' OpenSRF, I mean to have a situation
> where
> the OpenILS code runs with OpenSRF replaced by something simpler[2].
>
>
> [1] http://testing.evergreen-ils.**org/buildbot/waterfall<http://testing.evergreen-ils.org/buildbot/waterfall>
>
> [2] I have been investigating how OpenSRF can be stubbed out; it looks like
> some systemic code changes are needed in the OpenILS application modules in
> order for a simple stub to work.
>
> --
> steven chan
> schan at sitka.bclibraries.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20130528/ba7c7c21/attachment.htm>