(Taken From a recent post fo the software-testing email list)
It always amazes me when strong people come out and say publicly something that I have been mulling on for a few months.
James Bach’s comments on sapient processes do that for me. Let me give you the back story …
I do a lot of developer-facing test automation. That’s things like asserting the return values of a F-to-C temperture conversion function.
This has a lot of value to developers. It provides a functional spec to the maintenance programmer. It allows us (I am a tester/dev) to change the internals of the code and have some confidence that we didn’t break anything. It allows the developer to experience the API and change it to be easier to use.
Just about all of those are benefits to the developer – not the end customer. For the most part, they are design, maintenance, and software engineering benefits. Increasingly, I am wary of test automation that promises to “prove that the software works.” I find that on every release, the things that I need check are different – they are the things that have changed. Thus, the old test suite has less and less value as we add new functionality – and providing “hooks” and automation can be very expensive.
I think that Brian Marick’s Recent Comments echo this thread. Customer-Facing test automation isn’t all that great at, well, finding bugs, or proving that the software works.
Next month, I’ll be out in New York, trying to make this (and similar) points at the Google Test Automation Conference. Wish me luck …