Regression-Testing Redux

I know, I know, there are all kinds of testing, not just those two, how dare I forget about performance and security and so on. Here I am, being terrible, “just” talking about “functional” testing. Okay, fine, but not my point.

Today I just want to make two tiny distinctions, and explain why they are important.

It’s Not Just All Testing

One speaker talked about testing as if it were a static thing. We were failing as a community because manual testing wasn’t being replaced by automated testing. Imagine for a minute, though, what “success” of that goal would mean.

There you are, in a meeting, demoing the new middle name functionality. The VP of Customer success furrows her brow, thinking about the middle name function. She wonders aloud if you have two people with the same first and last name, but one has a middle and the other does not. What order will they appear in search results? She grabs the keyboard and begins to type. “WAIT!” you say. “STEP AWAY FROM THE KEYBOARD!“, for “this is an automated testing organization. We’ll put this test on the backlog, write it, and let you know what happens next sprint.”

Hopefully, no one would ever do that, because that is ridiculous. In that sense, “manual” testing will never go away. That investigation into how the software actually behaves in order to uncover new information – what humans are good at – is likely to stick around until AI can do it better. From what I understand of the state of AI, I’m not too worried about that.

This sort of exploration happens at many places in software, but the most common is either during story-development or just after, sometimes called a test step. This “story testing” is not regression testing. It is not checking that a change in one place caused the software to regress, or “break”, in another. It is just exploring this single change.

This kind of exploration is a valuable skill that not everyone has nor has an interest in developing. Any way of thinking about software testing should consider it.

Now let’s spend a minute on regression testing.

Regression Testing Methods

When people talk about regression testing, they often talk in terms of “End to End”, or the user journey. That is the whole path. For an e-commerce application, this is login, search, product page, add to cart, perhaps add a few things to cart. It also
includes checkout, sometimes called “path to purchase.” There are plenty of ways to perform regression testing, from checking each function to these sorts of End-to-End user functions. Some companies make “test cases”, and imply that “regression testing” consists of running all Priority1 and Priority2 “test cases.” To get to 100% percent test automation, they want to automate all the test cases.

Except computers are good at different things than humans. As Sauce Labs Solutions Architect Titus Fortner put it, you get the worst of both worlds. You lose the power of the human noticing something odd happened, and you fail to leverage the capability of the computer to work in parallel.

During SauceCon I got to spend some time with Josh Grant, another solutions architect at Sauce. In his tutorial, Josh introduced something new to me: Dom-To-Database, or D-To-D tests. In programmer terms, these tests are “full stack.” They go from the browser to the micro-service to the DB and back again, examining the results fully rendered in the browser. They are not, however, “end to end” in the classic testing sense. Instead of going through login, they might generate an auth token. Assuming some other test verifies navigation, they might go directly to the web page we need, do one thing, check the result, and get out. Each might use a different user with a unique id. Structure this way the tooling checks one essential piece of functionality, does it quickly, and does it in a way that is logically isolated from other checks. This makes debugging easy, makes fix/retest fast, and allows a team to run the entire suite in parallel in a handful of minutes.

Putting It All Together

All this just goes to demonstrate two points:

(1) Draw a distinction between what tooling and do, and what humans can do, and make it explicit, or else you’re in for a world of pain.

And the second rule, which is like unto the first

(2) If you are going to automate some part of testing, don’t start by assuming what the computer will do is like what the humans will do. Instead, aim for a synthesis of the two.

What are you doing to synthesize your testing?