When should a test be automated – I

I stand behind my last post on the Holy Grail, but it was often mis-interpreted as “no test automation.”

Now, certainly, that’s silly. At Socialtext, we use all kinds of tools to assist in our testing, some of them traditional run-capture-compare tools, others setup, bug tracking, grep, analysis, summary … the list goes on.

That leads to the question “when should a test be automated” – of course that’s a little bit silly, as having a computer look at one field, and having a human look at an entire window are two different things, but I do believe that it might be more helpful if I actually explored the area and provided some tangible advice.

To start with, let’s take a look at Brian Marick’s Article:

“When Should A Test Be Automated”, that he wrote in 1998

I will use that as a jumping-off point. So take a good look, and tell me what you think.

More to come.

3 comments on “When should a test be automated – I

  1. I’ve lived by Brian’s article since it came out. Still relevant, although now some of us have a whole development team to help us automate. My personal rule: If your team isn’t expert at automating, start with the assumption that you will automate this test. Then as you get into it if you find it doesn’t make sense, don’t. Otherwise, you will keep shying away from automating due to fear of unknown and steepness of learning curve.

  2. i can’t completely either agree or disagree. but one thing that sticks in my craw is:

    “My measure of cost – bugs probably foregone – may seem somewhat odd. People usually
    measure the cost of automation as the time spent doing it. I use this measure because the
    point of automating a test is to find more bugs by rerunning it. Bugs are the value of
    automation, so the cost should be measured the same way.”

    that seems to be missing the point of regression testing. i’ve been on too many projects where closed bugs have to be reopened later, such that the total cost of that bug has gone way up; had regression flagged it as soon as it was reintroduced, things would be cheaper. that’s one particular cost trade off i believe to be important.

Leave a Reply

Your email address will not be published.