[OPEN-ILS-GENERAL] Cucumber for QA

[Dan Scott]

On Tue, Feb 19, 2013 at 03:49:26PM -0800, James Fournie wrote:
> Hi there,
> 
> One of the suggestions I made with regard to QA in the recent IRC
> meeting, was to look into Cucumber for testing.   I thought I'd follow
> up a bit.  Caveat that this is not a QA silver bullet, just exploring
> some thoughts on it and how it could be used.
> 
> Cucumber is a Ruby-based tool for testing, based around
> behaviour-driven-development principles.  A key part of Cucumber, is
> the Gherkin language is uses.  Gherkin is a business-facing language,
> meaning it's designed to be understandable by non-developers.

<snip>
 
> As you can see, it's pretty simple to understand. My colleague Steven
> Chan in a Sitka team discussion noted that we do a lot of general
> testing in-house with each upgrade and it would be nice to capture
> that somehow.  I am sure that other Evergreen implementors have this
> knowledge drain as well and I think that Gherkin might be a way to
> capture some of that.
> 
> I've put together a basic Cucumber Ruby project for testing the TPAC
> mainly as a proof of concept.  Look in the 'features' dir at the
> 'feature' files for the Gherkin:
> 
> https://github.com/jamesrf/evergreen-ils-acceptance-test

This looks _awesome_, James. I've heard about Cucumber before but I
think my Perl/Python oriented brain basically ruled it out due to the
Ruby bit. But... that shouldn't matter for the purpose of testing. And
there is a _ton_ of what you've said that overlaps with the discussion
that was held during the hack-a-way. We had been struggling, a bit, with
how to capture the repeatable testing workflows and thinking about the
wiki as a repo, to start with, but there would be so much value to the
Cucumber approach.

An immediate win would be to extend your search examples to build on top
of the sample data we already have available to cover the kind of
searches that Thomas refactored with the QueryParser fix branch--to
ensure that we continue to get the results that we always wanted from
the previously working searches, and to ensure that we start getting the
results that we always wanted from the broken searches.

We need to move this beyond a proof of concept, this is a great concrete
way of moving the quality of Evergreen forward. Thank you for taking
things this far so far!