If you check Wikipedia, or google, you’ll likely come to the Consensus-version definition of Continuous Testing. That definition is fine, I guess, as far as it goes.
What it does not do in my book is provide enough of a litmus-test to say “Yes, this is, or is not, Continuous Testing.”
So I am proposing a definition with a little more meat on the bones. With something real, and not hand-wavy, people can disagree. That is okay. That is how progress has made.
I will say the term has come to mean only checks that run as part of a pipleine. At Excelon we can work with that, just as the term “Unit Tests” has come to mean automated checks that run in a tool like jUnit.
Here goes:
Continuous Testing is the process of executing setup, test execution, and teardown, of all automated checks. These tests run on a loop, for every version control change. Each change can be released to production without additional regression testing.
We worked to get that definition tight. As Anton de Saint-Exupery wrote “true perfection is achieved, not when there is nothing left to add, but when there is nothing left to take away.” We wanted each word to have maximal value. So, for example, the phrase “all automated checks.” In theory you could have some things you don’t run every time. Our Managing director, Matt Heusser, considered making that “of all run-every-time automated checks.” Any wider definition should include this greater term. It might be possible to come up with a better definition that covered more exception conditions Still, we found those three words just didn’t add enough value. The longer the definition, the more the actual message will be lost.
There are other infrastructure things you’ll likely need to do in order to eliminate regression testing. Multiple deploy points probably tops the list. With deploy points you can make a change to search with (some) confidence that the only feature at risk is search. Configuration flags can deploy dark then roll out to a limited user population or roll back. Extensive production monitoring and alerts advise the team of more problems earlier, before they can impact a large customer group. Observability makes it easy to debug complex problems in production. Those four, along with Continuous Testing, are mutually supportive, sort of like the old Twelve Practices of extreme programming. (If Matt gets the energy, he might pen something on Extreme Operations, or maybe not. The world doesn’t need a new manifesto, does it? For now maybe just call them the five cardinal virtues of DevOps. Now please don’t start and argument with me about the definition of virtue.)
For today, here’s a definition of Continuous Testing as it is being called and practiced that separates the wheat from the chaff.
Change our mind.
I liked that definition. I’d qualify the version control change as change in remote source control. I commit changes to my local git repo, I run unit tests & some integration tests but only before I push my changes to the remote repo.
I would be concerned having continuous delivery without continuous testing.
[…] Continuous Testing: Defined Written by: Matt Heusser […]
This definition is better, but I still say this is not continuous. T he word continuous in a math sense has a definite meaning. Here are some possible definitions:
{
ADJECTIVE
forming an unbroken whole; without interruption.
“the whole performance is enacted in one continuous movement”
synonyms:
continual · uninterrupted · unbroken · constant · ceaseless · incessant · [more]
forming a series with no exceptions or reversals.
“there are continuous advances in design and production”
mathematics (of a function) of which the graph is a smooth unbroken curve, i.e. one such that as the value of x approaches any given value a, the value of f(x) approaches that of f(a) as a limit.
}
I get that this looks like an entire cycle, but once you tear down you have a break in testing. That’s why I call this Discrete testing. Continuous would imply little to no interruption between test runs.