[OPEN-ILS-DEV] Putting the community's QA money where our dev mouth is

[Kathy Lussier]

I'm guessing this message was sent mostly to get developer feedback on 
including unit tests. However, since there has been no developer 
feedback and since this e-mail raised a lot of questions for me, I'll 
jump in with those questions.

Overall, I support better QA practices and have been following much of 
the QA discussion over the past few months. However, after reading this 
e-mail,  I realized I don't have a good understanding of how these tests 
lead to better quality. Since MassLNC sponsors a lot of development 
projects, I think it's important that I have a better understanding of 
these tests since it may need to be something that we might want to 
consider including in future development contracts.

Can somebody provide an end-user explanation (please don't be afraid to 
dumb it down as much as possible) as to how pgTAP tests help with the QA 
process? In looking at the QA report, I see:

> The idea here is that if we can get a commitment from the core 
> developers, we can use this script once to create a base set of tests 
> that we can then commit to the master branch. From then on we can 
> simply maintain and modify those tests whenever we create a database 
> upgrade script. These tests can then prove that both the upgrade 
> scripts and the seed files do the same thing, and folks performing 
> upgrades can sanity-check their work to ensure that they didn't miss 
> invoking an upgrade script, or that an upgrade script somehow failed 
> unnoticed.

Is the intent of the test to verify that that database changes make it 
into the upgrade scripts? Is it also doing some kind of check to make 
sure the database change doesn't break anything? Is the test only useful 
for that one database change or is it something that is then part of a 
suite of tests that are run whenever new database changes are added?

In looking at the QA report, I think I have a better handle on how 
integration tests work. Basically, it looks like we have some scripts in 
Evergreen that performs various functions, and the idea is to run those 
tests when new code is added to make sure we continue to get expected 
results. Is that assessment correct? The recommendation to continue to 
add integration tests, then, is to make sure that any new functionality 
be included as part of that testing to ensure future code changes don't 
break that functionality, right?

I'm curious about the time/effort required if the recommendations from 
the QA report were adopted. For those who have created pgTAP or 
integration tests, how much of a time commitment do you think would be 
required to come up to speed on creating these tests? Once developers 
are up to speed on adding these tests, do you have any sense of how much 
time it would add to the development process to include them?

A while back, there was some talk on the list about using Cucumber for 
testing. http://markmail.org/message/otvljkdd4pwtg2ov My understanding 
from the discussion thread was that it would make it easier for 
non-developers to add tests. Is that something that could be used to 
help the community ease into qa practices?

Thank you all for your patience as I try to gain a better understanding 
of these issues.

Kathy

Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 343-0128
klussier at masslnc.org
Twitter: http://www.twitter.com/kmlussier

On 11/1/2013 5:01 PM, Dan Scott wrote:
> In the QA report for which several members of the community generously
> paid $30,000, the "Moving Forward" section at
> http://nox.esilibrary.com/~jason/qareport/qa.html#_moving_forward
> states:
>
> "We recommend that the development community start including integration
> tests with their changes to the backend, and pgTAP tests with their
> database changes (there was discussion and general interest in this
> during a developers meeting)."
>
> The referenced developer meeting is minuted at
> http://evergreen-ils.org/meetings/evergreen/2013/evergreen.2013-08-27-14.04.html 
>
> with the agreement "general interest in easing into qa practices
> demonstrated by phasefx, phasefx to hold hands with everyone, especially
> through the mailing list"
>
> I'm concerned that since then we have seen a number of database changes
> without corresponding pgTAP tests (the most recent pgTAP test was
> committed Sept 3rd, while there were a ton of changes committed through
> October). I did bring this up in the IRC channel with respect to one of
> the recent bug fixes that was committed for a fairly fundamental
> function, but my gentle prod in that case appears to have been
> overlooked.
>
> I recognize there is pressure to get the 2.5 release out, but if we
> continue to follow our past approach of not including unit tests where
> the path has been blazed for us when we commit changes to the database,
> our quality assurance is going to assuredly be similar to the quality we
> have produced in the past.
>
> In short, we have heard strongly from the community that we are not
> producing software of the quality that they expect; and the community
> has followed up those words with a significant investment in the QA
> project; and we as a developer team are _not_ following through with the
> process improvements we had agreed to adopt.
>
> FWIW, should anyone want to follow some commits to teach themselves how
> to add pgTap tests, I did go through the learning process to support
> https://bugs.launchpad.net/evergreen/+bug/1242999 - the tests I created
> and the instructions for running them are basic, but they're a start.
> Thanks to Galen for giving me a pointer in the right direction.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20131105/c0fb18c5/attachment.htm>