Make Software Testing Great … Again?

Yesterday, as I was attending James Bach’s course reinventing testers, I took a picture of a slide that I knew would be contentious in the Software Testing world … and still tweeted it.

The responses were as acrimonious as the original text, and my response, which boiled down to “hey man, it’s not my slide” fell short. Jared Quinert asked me if I didn’t endorse the text, why tweet it? That is a reasonable question, but twitter is wrong medium for it.

So I decided to write this blog.

It’s easy to dismiss the entirety of the slide, as Augusto Evangelisti did – so let’s start where we might have the most agreement, work backwards to intent, and then draw some conclusions.

Just the facts Ma’am

My understanding of the Agile Fusion conference is that it went down pretty much as described. When I initially wrote this post, I was not aware of anyone who attended that conference that disputed the key events(*). Brian did go focus on Agile-Testing and no longer presented himself as CDT shortly after that conference.

My first encounter with what would later be called “Agile” was through Scrum and Extreme Programming. The book XP:Installed really does state there is no test role (though it is worded as “no QA department”, something I agree with). When I read XP:Installed and skimmed XP:Explained (“The White Book” above), I thought of myself as a programmer. I remember explaining Test Driven Development by saying aloud “Testing is boring but coding is fun”; I said this often enough that my colleague, Tessa Welsh Oxner, once cut me off and completed my sentence for me.

I’m pretty sure I got that idea from somewhere.

Regarding the programming specialists bullet. If I recall correctly, the book XP:Installed, which was the first book I ever read about “Agile”, explicitly says whole team of programmers.  I have been told multiple times at the Agile conference that all team members “should” be programmers. In 2013, the example was that “In the Marines, everyone leaves boot camp a qualified infantry solider first, then gets a specialty.” Two issues you might take with my experience are (1) Agile is a “big tent”, so those opinions are not representational, (2) I (Matt) don’t really take much exception to the example, if one actually studies how the Marines work. Waterdogs are not really expected to be qualified infantry soldiers, but they do have a basic knowledge of concepts that could be helpful in an emergency.

At this point, we can not longer dismiss the slide entirely – it has some facts in it. You can take issue with the tone. I understand that, and want to talk about it. For now, let’s keep going.

Digging A Bit Deeper

Here’s an excerpt from a email I got a few years ago from Lisa Crispin about the early Agile reaction to testers as a role. The story is about Ron Jeffries, who was on the first XP team and is a co-creator of the Agile Manifesto. He is a co-author of XP:Installed:

–——— Forwarded message ———-
From: Lisa Crispin <lisa.crispin@gmail.com>
Date: Thu, Sep 24, 2009 at 8:21 PM
Subject: [writing-about-testing] Re: Touchy Topic-Blog Post?
To: writing-about-testing@googlegroups.com

Ron Jeffries once told me to my face (this was back in 2000 or 2001) that anyone on an XP team who doesn’t write production code is a useless fifth wheel, which made me go home and cry about not being able to work on an XP team. I got over this quickly as I was already working on an XP team and clearly adding value. Ron changed his opinion and even helped us with our first book, Testing XP. And, some of Ron’s views changed my opinions on some aspects of testing.

I have had tens of thousands of words of correspondence with Ron, through which I believe we came to a grudging understanding of each other. It took some time.

Now a tough bullet point: The book by Lisa Crispin and Janet Gregory. It doesn’t really discuss test design techniques.  Let’s talk about why this might be, and whether or not that is okay. To do so, we need to take a step back.

In the Bad Old Days

Prior to the Agile movement, we had a movement that believed in documenting everything and minimizing face-to-face, feedback based communication. (That’s a quick, lossy summary based on the literature review I did in graduate school, but I believe a fair one. I could defend it in a different post if you’d like).

Back in those days, on a bad day, the testers would have no access to the customer, or the analyst, or the programmer. Instead, we’d slide something like a specification under the wall along with a build, and tell testers to “go forth and test.”

In order to get good at their jobs, testers needed to develop skills. We learned how to make decision trees, we learned common failure modes for our platforms, we learned combinatorics and pairwise techniques and quick attacks. We studied soap opera tests and other methods to cover the highest amount of code and input space in the least possible time. (I say we lightly; I was a programmer in teams that did not have testers, and I studied testing as craft along with code as craft. From time to time I would test other people’s code because I was interested in it. My first programming assignment is something that today we would call model-driven testing. Back then we called it an “electronic audit.”)

We developed these skills, at least in part, at least in my opinion, to compensate for the sad working conditions.

Luckily, many leaders across software delivery were equally appalled at the dehumanizing conditions that were the norm, and the Agile Manifesto was born.

Enter XP and Agile

First, imagine shifting from the extreme above (most of the time it wasn’t quite that bad) to a world where you put all the technical staff in the same room and everyone talks to each other every day. The programmers are doing pair programming, Test Driven Development, and refactoring, so first time quality is higher. The team is also “kicking off” stories creating a shared mental model of what will be built with hard, detailed examples, so first time quality is higher. Some large amount of the acceptance checks are automated. The programmers study code as craft, so they have fewer regression errors, because they are isolating components, not repeating themselves, and so on. Make rollback and fix much cheaper, add a demo so the customer can ask questions, and add some time “playing with it” doing informal exploration, and the risks that classic test craft/excellence addresses are markedly decreased. Everyone takes an interest in quality as a whole team, including the product owner and customers.

Teams using this formula often do okay without formally studying testing, and that might be okay.

Given that system of forces, it’s not a huge surprise that the book by Janet and Lisa doesn’t discuss test design techniques. (By which I mean: Given this particular software, what questions would you ask to learn about the quality?)

I see Janet and Lisa’s contribution being the “Agile” part, specifically working collaboratively, including the shift to everyone owns quality. After all, the “ground” of test design had already been covered well by others. It is hard for me to state in a blog post like this, with a different focus, what a big deal the work of Janet and Lisa has been. It is a non-trivial change in attitude and behavior that air-quote “Agile” teams fail to make. I feel strongly enough about this that I felt compelled to write about making the change,  including the culture change, in a book I am writing called Save Our Scrum, along with Markus Gaertner.

Lisa and Janet’s work covers this area for testing, covers it well, and doesn’t talk about much else, which leaves them open to the critique that agilists “don’t study test [design]ing.”

But Do “Agilists” Study Software Testing?

As I mentioned earlier, “Agile” is a big tent. If you search the agile test literature, you will find some – some of which I wrote, and earned awards (2014) for (2015). Elisabeth Hendrickson, widely associated with Agile Software, wrote a book on exploratory testing that does cover test design.

Personally, I am widely associated with Agile-Testing, and not offended by the claim. I think the “collaboration stuff” plus testing skills are like peanut butter and chocolate. If anything, as someone widely associated with Agile Techniques and Software Testing, I take it as a mild personal critique, as, within the Agile community, I am one of the “test design” folks. That is a large part of my role in the community, and I have not done enough. James has told me this privately in the past.

I accept his critique as valid, even if I do not appreciate the tone.

I could go into some depth on the TDD/BDD slide, but I think it suffices to say this is a common over simplification, trap, and risk that many teams fall into, and some advocate as good practice.

Now on to the disagreements.

Matt’s Disagreements With The Slide

I do not feel ‘under attack’ as a tester. When my friend Jessie Alford says that “testers go to war every day”, that does not resonate with me. I do think there was a divide between ‘Agile’ and ‘Testing’ in the early days – while I was speaking about Agile-Testing in the early 2000’s at testing conferences, I was not attending ‘Agile’ Conferences, and did not feel part of the physical, in-person community.

By the time I went to the Agile Conference, I believe Lisa and Janet had sort of  ‘paved the way’ for me – I never experienced anything like that email from Lisa above, and I am grateful for it.

But What About the Tone?

The tone of the slide is hyper critical. It doesn’t seem to seek to understand. It is more than a little judgy. It certainly isn’t my style. If you’ve read this far, however, I believe I have made the case that the points in the slide are worth thinking about, making it worth tweeting. Or, perhaps, due to the sensitive nature of the material, worth a blog post.

Fair enough, I tried.

Let’s keep talking.

UPDATE 5/19/2016:

Brian Marick disputes the Agile Fusion Conference events but did not provide an alternative narrative. Andy Tinkham does not dispute it, in that he remembers conflict and no narrative. Bret Pettichord, who was also there, points out the year in the slide is off by one.

I have made a sincere attempt at an unbiased search for truth here, and the situation bothers me a little as a journalist. Bret did me the great favor of pointing to Andy Tinkham’s notes from the Agile Fusion Conference, which you might find helpful if you’d really like to do some archeology.