Pretend for a moment you are a new development manager or director.
You hired a programmer to lead a test automation effort. Gave the programmer time, resources, and attention. The programmer has no responsibilities outside of the tooling. The automation has had time to mature. It was supposed to speed you up.
Now imagine this conversation, at 10:00AM.
Manager: “I’d like to go to production at noon. Can we do that?” Programmer: “I don’t know, we need to run the automation.”
Manager: “Okay, sure, get to it.
”Programmer: “First we need to get the exact build we need to run on staging.”
Manager: “Fine, do that, then run the tests.”
Programmer: “Then I need to set up the test data.”
Manager: “Okay, fine, whatever, do it.”
Programmer: “Then I can run the tests.”
Manager: “Fine, so you’ll be done by noon?”
Programmer: “Probably. Then we’ll have failure results to check.”
Manager: “You always have failures?”
Programmer: “Not real failures. Sometimes the tests are flaky or there are environment problems. Sometimes the User Interface has changed but the tests haven’t kept up. So I need to do some maintenance work.”
Manager: “So you’ll know in the early afternoon?”
Programmer: “Sure.”
… time passes …
2:00PM – Automation Delay
Manager: “Okay, what’s up?”
Programmer: “Bugs. Just a couple of showstoppers.”
Manager: “So, no ship today?”
Programmer: “The production programmer said he’ll have it fixed by 4:00PM.”
Manager: “That’s great! We can ship by end of day?”
Programmer: “No, I need to rerun the tests.”
Manager: “Okay, but you can do that overnight, right?”
Programmer: “No, we really need someone to baby-sit them. The screen could lock, that sort of thing. I’ll kick it off in the morning.”
The next morning, at 10:00AM, the new build is certified and can get to production.
Let me be clear:
When the director invested in test automation, that is not what he thought he was buying.
I call this process of fits and starts “automation delay.” It commonly happens when test groups, left to themselves, decide to automate the process, especially though the User Interface.
Interlude: Getting Started
A few years ago I attended training from a major tool vendor. The idea was that some of us independent consultants could learn the tool and support it. After learning how to use the tool, I started to ask questions about a command-line runner and the output the tool produced — so we could kick it off on a
schedule, capture the output, then decide if the checks passed, ideally by a computer, that could then choose if the software should be promoted to the next level.
We spent a little time tracking down in the documentation that yes, a command- line runner did exist, and the output was predictable, so we could wrap it in something that produced output for a continuous integration (CI) server to parse.
Once we proved it in theory, the trainer moved on. “After all”, he said “Most of you will be called into companies just getting started. No need for this advanced stuff.”
Just getting started.
So set up the test server by hand, seed the test data by running an import or restore database of some type, then kick off the automation to run at your desk.
Better run it at lunch, or at least find something else to do while the driver runs.
Yuck.
Worse, this scenario sets us up for the problem I mentioned at the beginning – it sets up the customer for automation delay.
The Bottom Line
Too many testers start out automating without support from anyone else. All they see is the GUI, so they begin automating the GUI. That leaves all the problems we mentioned before, and leads to automation delay.
Don’t do that.
Instead, start with the running the tests under continuous integration, which probably means extending CI to actually create a virtual machine and install the latest build, before running the automated checks. Good. Do that. That will probably be hard, and you might need the support of Dev or Operations. That is okay; that is sort of where the term DevOps comes from.
Of course, what I wrote above involves a great deal of sweeping generalizations. You could be anywhere on the spectrum; you might make batch applications that don’t really have a User Interface. For the time being, consider the impact of automation delay, if it could be happening to you — and what to do about it.
[…] Automation Delay – and how to prevent it – Matt Heusser – https://xndev.mystagingwebsite.com/2021/02/automation-delay-and-how-to-prevent-it/ […]
[…] Automation Delay – and how to prevent it Written by: Matt Heusser […]
[…] two years, that could be a thirteen-hour delay. That amounts to 1.5 business days. I call this automation delay, and that is a major source of failure of test tooling projects. It fits one definition of scaling, […]