What’s a "Test Framework"?

Shrini Kulkarni has been after me to define my terms; after all, I keep writing about “Test Frameworks” but I’ve never defined the term.

Wikipedia defines a framework as “A basic conceptual structure used to solve a complex issue. It also warns that “This very broad definition has allowed the term to be used as a buzzword.”

When I use the term, I mean any support, infrastructure, tool or “scaffolding” designed to make testing easier, and (often) automated.

For example: Let’s say you have a simple program that converts distance from Miles to Kilometers. The application is a windows application. Every time we make a change, we have a bunch of tests we want to run, yet we can only enter one value at a time, manually. Bummer.

Yet we could think of the software as two systems – the GUI, which “just” passes the data from the keyboard into a low-level function, and “just” prints the answer, and the conversion formula function.

If we could somehow separate these two, and get at the formula function programmatically. Imagine that the formula function is struck in a code library, which can be shared with many different programs.

Then we could write a “tester” program, which takes an input file full of input values and expected results. The “tester” program simply calls the library, compares the actual result to the expected, and prints out a success or failure message.

This is basically how I test a lot of my code, using a program called Test::More. You could call Test::More and it’s friends (Test::Harness, and so on) a “framework.”

We can go dig into the details and test at the developer understanding level, or bubble up and only test things that have meaning to the customer. One popular framework for these higher level, business logic tests is FIT/Fitnesse

Of course, there is more to the application and just the business logic. The GUI could accept the wrong characters (like letters), format the decimals incorrectly, fail to report errors, handle resizing badly, or a half dozen other things. Even with one “framework”, we still have the problem of testing the GUI (not to be forgotten) and testing the two pieces put back together again – “Acceptance” testing, or, perhaps, “System” testing.

This “outer shell” testing can also be slow, painful, and expensive, so there are dozens of free/open or commercial testing frameworks that allow you to ‘drive’ the user interface of windows or a web browser. With the big commercial tools, people often find that they are writing the same repetitive code, over and over again, so they write libraries on top of the tool like Sifl.

The big dog web browser automation systems are Selenium and Watir.

Years ago (back when he was at Microsoft), Harry Robinson once told me that MS typically had two types of testers: Manual Testers, and Developers who like to write frameworks. The problem was that the Developers would write frameworks that no one was interested in using. His assertion (and mine as well) is that people who straddle the middle – who like to test, and like to write software to help them test faster, can be much for effective than people entrenched on either side.

Thus, you don’t set out to write a framework – instead, you write a little code to help make testing easier, then you write it again, then you generalize. Over time, slowly, the framework emerges, like Gold refined through fire.

But that’s just me talking. What do you think? (And, Shrini – did I answer your questions?)

6 comments on “What’s a "Test Framework"?

  1. Thus, you don’t set out to write a framework – instead, you write a little code to help make testing easier, then you write it again, then you generalize. Over time, slowly, the framework emerges, like Gold refined through fire.

    This is how many things get added to our GUI testing frameworks. We generally first write custom automation code for specific needs. Then, if we have to do it again in another context, we consider modifying the context-specific code to work in other contexts and make it part of our existing framework. Over time, our framework — our toolbox — grows.

    The first version of my model-based test engine was very simple. Over time it has grown to make test development easier. Most of the added features started as quick and dirty tools built for testing specific functionality of specific applications.

    Ben Simo

  2. An interesting take on “your write a little code and write and generalize”. The incremental hacks help in solving your immediate needs. However the changes may not many not lead to generalization of the code to a framework.
    My initial experience with incremental hacks to the original test code has often ended up with unmaintainable code yet something that meets all your immediate needs.
    Its only after a number of failures and experiences you learn the art of transforming the code to a useful reusable framework.

  3. When Wow Gold wolf finally found the wow gold cheap hole in the chimney he crawled cheap wow gold down and KERSPLASH right into that kettle of water and that was cheapest wow gold the end of his troubles with the big bad wolf.
    The next day the Buy Wow Goldlittle pig invited hisbuy gold wow mother over . She said “You see it is just as mygamegoldI told you. The way to get along in the world is to do world of warcraft gold things as well as you can.” Fortunately for that little pig, he buy cheap wow gold learned that lesson. And he just k4gold lived happily ever after!.

Leave a Reply

Your email address will not be published. Required fields are marked *