[OPEN-ILS-DEV] Putting the community's QA money where our dev mouth is

[Kathy Lussier]

Thank you Jason! Your explanations were very helpful.

Kathy

Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 343-0128
klussier at masslnc.org
Twitter: http://www.twitter.com/kmlussier

On 11/5/2013 12:59 PM, Jason Etheridge wrote:
> On Tue, Nov 5, 2013 at 11:53 AM, Kathy Lussier <klussier at masslnc.org> wrote:
>> I'm guessing this message was sent mostly to get developer feedback on
>> including unit tests. However, since there has been no developer feedback
>> and since this e-mail raised a lot of questions for me, I'll jump in with
>> those questions.
> Kathy, I took Dan as trying to rally developers to actually move
> forward with more testing.
>
> <snip: automated testing>
>> it may need to be something that we might want to consider including in
>> future development contracts.
> I think that's a grand idea.  Equinox promises to do this with all of
> its new development projects.
>
>> Can somebody provide an end-user explanation (please don't be afraid to dumb
>> it down as much as possible) as to how pgTAP tests help with the QA process?
>> In looking at the QA report, I see:
> The particular part you quoted is about a test generator for creating
> a baseline set of tests.  This hasn't really been discussed widely,
> and may or may not be a good idea.  It's orthogonal to the idea of
> using pgTAP in general.  pgTAP tests will help just like other tests,
> which I think you have a handle on below.  The distinction between
> unit and integration tests isn't that important; what's important is
> coverage and quality of the tests, and the actual process of writing
> the tests.  Automated tests help us catch undesired changes in
> software behavior.  And the process of actually writing tests can
> change how development is done, improving the development process by
> changing how developers think about their work, even if a specific
> test never actually catches a bug.
>
>> In looking at the QA report, I think I have a better handle on how
>> integration tests work. Basically, it looks like we have some scripts in
>> Evergreen that performs various functions, and the idea is to run those
>> tests when new code is added to make sure we continue to get expected
>> results. Is that assessment correct? The recommendation to continue to add
>> integration tests, then, is to make sure that any new functionality be
>> included as part of that testing to ensure future code changes don't break
>> that functionality, right?
> Yes and yes.
>
>> I'm curious about the time/effort required if the recommendations from the
>> QA report were adopted. For those who have created pgTAP or integration
>> tests, how much of a time commitment do you think would be required to come
>> up to speed on creating these tests? Once developers are up to speed on
>> adding these tests, do you have any sense of how much time it would add to
>> the development process to include them?
> For my part, it was mostly a mental barrier.  Developers write code,
> and tests are simply more code, and not a whole lot in comparison to
> what is being tested.
>
>> A while back, there was some talk on the list about using Cucumber for
>> testing. http://markmail.org/message/otvljkdd4pwtg2ov My understanding from
>> the discussion thread was that it would make it easier for non-developers to
>> add tests. Is that something that could be used to help the community ease
>> into qa practices?
> I think the concepts there could be very useful.  I want to look at this more.
>
> Thanks Kathy,
>