[OPEN-ILS-DEV] Putting the community's QA money where our dev mouth is

[Jason Etheridge]

On Tue, Nov 5, 2013 at 11:53 AM, Kathy Lussier <klussier at masslnc.org> wrote:
> I'm guessing this message was sent mostly to get developer feedback on
> including unit tests. However, since there has been no developer feedback
> and since this e-mail raised a lot of questions for me, I'll jump in with
> those questions.

Kathy, I took Dan as trying to rally developers to actually move
forward with more testing.

<snip: automated testing>
> it may need to be something that we might want to consider including in
> future development contracts.

I think that's a grand idea.  Equinox promises to do this with all of
its new development projects.

> Can somebody provide an end-user explanation (please don't be afraid to dumb
> it down as much as possible) as to how pgTAP tests help with the QA process?
> In looking at the QA report, I see:

The particular part you quoted is about a test generator for creating
a baseline set of tests.  This hasn't really been discussed widely,
and may or may not be a good idea.  It's orthogonal to the idea of
using pgTAP in general.  pgTAP tests will help just like other tests,
which I think you have a handle on below.  The distinction between
unit and integration tests isn't that important; what's important is
coverage and quality of the tests, and the actual process of writing
the tests.  Automated tests help us catch undesired changes in
software behavior.  And the process of actually writing tests can
change how development is done, improving the development process by
changing how developers think about their work, even if a specific
test never actually catches a bug.

> In looking at the QA report, I think I have a better handle on how
> integration tests work. Basically, it looks like we have some scripts in
> Evergreen that performs various functions, and the idea is to run those
> tests when new code is added to make sure we continue to get expected
> results. Is that assessment correct? The recommendation to continue to add
> integration tests, then, is to make sure that any new functionality be
> included as part of that testing to ensure future code changes don't break
> that functionality, right?

Yes and yes.

> I'm curious about the time/effort required if the recommendations from the
> QA report were adopted. For those who have created pgTAP or integration
> tests, how much of a time commitment do you think would be required to come
> up to speed on creating these tests? Once developers are up to speed on
> adding these tests, do you have any sense of how much time it would add to
> the development process to include them?

For my part, it was mostly a mental barrier.  Developers write code,
and tests are simply more code, and not a whole lot in comparison to
what is being tested.

> A while back, there was some talk on the list about using Cucumber for
> testing. http://markmail.org/message/otvljkdd4pwtg2ov My understanding from
> the discussion thread was that it would make it easier for non-developers to
> add tests. Is that something that could be used to help the community ease
> into qa practices?

I think the concepts there could be very useful.  I want to look at this more.

Thanks Kathy,

-- 
Jason Etheridge
| Support Manager
| Equinox Software, Inc. / The Open Source Experts
| phone: 1-877-OPEN-ILS (673-6457)
| email: jason at esilibrary.com
| web: http://www.esilibrary.com