A Testing Challenge

As a fan of incremental/iterative methods, I like the idea of test automation.

In theory, everything should be retested every release, but with two-week iterations, that simply is not going to happen. With test automation, we can at least have some confidence that the software didn’t introduce any major regression errors outside of the obvious features being changed.

So, it would be really nice to have a computer with a big button that says “test” that runs all the tests and comes back with a green (or red) light. For unit tests, I use these in spades.

The problem is customer acceptance tests. There are some tools for automating acceptance tests: Most notably FIT and Fitnesse.

Fit and Fitness take a set of inputs and compare that against a set of outputs. To do this, they have ‘fixtures’ that call into the application, call a function, get the results, and compare that to output. Since Fitnesse is written in Java, it can have ‘hooks’ into your application — if that app is also written in Java or a language which FitNesse Supports.

This can work for standalone applications; logical silos that take an input, transform it, and provide an output.

Now the challenge:

—-> Lately I’ve been working with IS Shops, not software companies. These are organizations that support a traditional, brick-and-mortar business. Instead of producing standalone apps, these organizations are more often integrating two applications.

The software that is being testing isn’t the app itself (that ws commercial-off-the-shelf) but the data bridge between the apps.

For example, one application pulls data out of an ERP system, stores it as a flat file, and imports it into a financial system. The requirements are “Make the Financial System LOOK LIKE the ERP System for accounts A, B, and C.”

Another application pulls data out of the ERP system and populated the data warehouse. A third creates a flat file which is sent over to a trading partner “Make sure the trading partner knows all our gold members, the member ID, and eligibility dates” are the requirements.

Think about it – the requirement is to take one set of black-box data, and import it into another black box. We can test the data file that is created, but the real proof is what the second system accepts — or rejects.

And, no offense, but for some of these Apps, FIT isn’t a very good fit.

First of all, the test databases used are refreshed every three months from production. That means that you either have to find test scenarios from live data (and hope they don’t change next time you refresh) or re-enter every scenario in test every three months.

Now, take the trading partner example. The best you can do within your organization is to test the file. The interface might take three hours to run, then you GREP the file for results and examine. You’ll have to write custom fixtures to do this, and your programming language isn’t supported by FitNesse. Or you could write a fixture that takes a SELECT statement to count the number of rows that are generated by the file, run the interface, and compare.

Of course, a programmer is going to have to write the SELECT statement. Is it a valid acceptance test?

Or you could have the number of rows fixture be approximate – “Between 10,000 and 15,000” – customers could write this, and it guarantees that you didn’t blow a join, but not much else.

You could write code that accesses the deep guts of the application, turning it sideways to generate a single member at a time, thus speeding up the acceptance test runs to a few seconds. That’s great for the feedback loop, but it’s more of a unit test than an acceptance test.

You could suggest I re-write the whole thing to use web services, but that introduces testing challenges of an entirely different kind. To be frank, when I have a problem and people suggest that I re-write the whole thing without recognizing that it would present an entirely different set of challenges, it’s a sign to me of naiveté.

I submit that all of these would be a significant investment in time and effort for not a whole lot of value generated.

So, I still want to write customer acceptance tests, but I’m not sure this is the right paradigm to do it. I also have a handful of tricks and techniques I have used over the years to make this easier. I will reveal them in a future post, but in the mean time, here’s my challenge:

What would you suggest to solve this puzzle?

I should add that I don’t think this is a trivial puzzle; at least, more than half of the people of which I ask this give an answer that I believe to be unsatisfactory. Can you do better?

13 comments on “A Testing Challenge

  1. As far as I’m concerned, the Bible for this kind of work is Gregor Hohpe’s work at http://eaipatterns.com/.

    In particular, check out http://www.enterpriseintegrationpatterns.com/docs/TestDrivenEAI.pdf

    I’ve taken two approaches to this kind of work:

    First, have tests that validate that any input to the system is processed correctly in the output. For instance, check that input Field 22 appears in the filename, and input Field 87 appears in the output file header, or whatever. This approach is amenable to risk-based analysis.

    Second, account for the possible range of input data. Sometimes this means selecting records semi-randomly from the input system. Sometimes it means generating input data using semi-random methods.

    Finally, run a LOT of tests. Run tests all night, run tests all week, run tests all month. Harry Robinson’s work is a good reference for this sort of thing.

  2. If the live data refreshed from production is exhaustive and you cannot find a test scenario then you may technically have a bug in the code but you will not hit the bug with the current production data (you may hit it in the future with new production data.) Now you can wonder “if a bug exists in the code but doesn’t make any noise did the tree really fall?”

    One approach – get the live data from system A, run the data transformation/import into system B in a test environment, and get users from systems A and B in a room together to discuss whether system B properly represents the data from system A. The discussion should at least address some normal scenarios and several flavors of extreme/weird scenarios.

  3. Few things to test apart from what are already mentioned,
    -If A is pumping out the data at faster pace than speed with which B takes-in, in that case there should be proper queueing logic in the bridge.
    -If suddenly B goes down then will the data pumped in by A is queued.
    -If bridge itself is not able to process the data at speed with which A pushes in.

    Code Inspections

  4. When faced with this sort of problem I usually try to reverse engineer the input files from the output files and compare. That way I can find fields and records that are missing, extra stuff, bad transformations, etc. Of course, this isn’t always possible but it’s a good approach if you can do it. You might have to extract information from the target system to check that the import was complete.

    Integration testing and architectural-level issues are are a whole different kettle of fish and something that the agile guys neatly avoid discussing. Unit tests aren’t the answer to everything.

  5. If I understand, you have three different processes taking data out of one system and placing it into three different systems/formats:

    1) Another application
    2) A database
    3) A flatfile

    The requirement to test is that the data in all three destination systems is the same as displayed by the source system.

    Do I have this right?

  6. First of all: Great Blog! You really inspired me to learn more, thank you!
    Would you say that Cubes are an option to solve this?
    Once again, thanks for your contributtion to the testing world.
    Best Regards, André Vondran

  7. You said you will love me wow gold the whole life, but WoW Gold you marry her. You said you will wow power leveling,come to marry me, but this will not be carried out forever.WoW Gold I am trying my best to forget you and do not love you anymore. wow leveling But I failed and I still love you. Maybe wow leveling she needs you more compared wow leveling with me. So I tell you that world of warcraft power leveling you should love world of warcraft power leveling her and take good world of warcraft leveling care of her. You said I was so kind.world of warcraft leveling Yes, because I love you,world of warcraft leveling I hope you will be power leveling happy forever.

  8. http://www.pumachaussure.com/ : puma
    http://www.pumachaussure.com/ : femme puma
    http://www.pumachaussure.com/ : homme puma
    http://www.pumachaussures.com/ : chaussures puma
    http://www.pumachaussures.com/ : puma basket
    http://www.pumachaussures.com/ : puma
    http://www.pumachaussures.com/ : puma basket chaussures
    http://www.pumachaussures.com/ : basket puma
    http://www.uggsoutlet.us/ : UGG Online Store
    http://www.uggsoutlet.us/ : cheap ugg boots
    http://www.uggsoutlet.us/ : black ugg boots
    http://www.uggsoutlet.us/ : ugg boots on sale
    http://www.uggsoutlet.us/ : cheap uggs
    http://www.chiflatiron.us/ : chi flat iron
    http://www.chiflatiron.us/ : chi flat irons
    http://www.chiflatiron.us/ : chi hair straightener
    http://www.chiflatiron.us/ : chi ceramic flat iron
    http://www.chiflatiron.us/ : chi iron
    http://www.thehairstraighteners.com/ : cheap hair straightener
    http://www.thehairstraighteners.com/ : hair straighteners
    http://www.thehairstraighteners.com/ : chi hair straightener
    http://www.thehairstraighteners.com/ : chi hair straighteners
    http://www.thehairstraighteners.com/ : chi flat iron
    http://www.northfacejackets.us/ : north face factory outlet
    http://www.northfacejackets.us/ : north face outlet store
    http://www.northfacejackets.us/ : discount north face jackets
    http://www.northfacejackets.us/ : north face jackets
    http://www.northfacejackets.us/ : spyder jackets
    http://www.northfacejackets.us/ : cheap north face jackets
    http://www.jacketscart.com/ : cheap north face
    http://www.jacketscart.com/ : north face outlet store
    http://www.jacketscart.com/ : discount north face jackets
    http://www.jacketscart.com/ : cheap north face jackets
    http://www.jacketscart.com/ : spyder jackets

  9. http://www.chihairstraightener.us/ : chi hair straightener
    http://www.chiflatiron.us/ : chi flat iron

    http://www.polool.com/ : new polo shirts

    http://www.pursecart.com/ : cheap handbags
    http://www.pursecart.com/ : cheap bags
    http://www.pumachaussures.com/ : puma chaussures
    http://www.pumachaussures.com/ : chaussures puma
    http://www.pumachaussure.com/ : chaussure puma

    http://www.jacketsworld.com/ : Men's North Face
    http://www.jacketscart.com/ : Women's North Face

    http://www.myhairstraighteners.co.uk/ : hair straighteners
    http://www.sexylingeriestore.us/ : sexy lingerie store
    http://www.uggsoutlet.us/ : cheap ugg boots
    http://www.12hot.com/ : tattoo wholesale
    http://www.polooo.net/ : men's clothing
    http://www.acrazystore.com/ : women's clothing

    http://www.thehairstraighteners.com/ : cheap hair straighteners
    http://www.polosales.com/ : cheap Lacoste polo shirts
    http://www.superpolos.com/ : cheap Lacoste polo shirts
    http://www.shoesbuying.com/ : 2009 nike shoes
    http://www.shoesbuying.com/2009-nike-shoes-c-153.htm : new nike shoes
    http://www.shoesbuying.com/air-max-classic-bw-c-185.htm : Women's max
    http://www.shoesbuying.com/mens-max-93-c-188.htm : Men's max 93
    http://www.shoesbuying.com/nike-shox-c-165.htm : nike shox
    http://www.shoesbuying.com/nike-force-c-174.htm : Nike air force
    http://www.shoesbuying.com/nike-2003-c-181.htm : Nike air max 2003
    http://www.shoesbuying.com/nike-air-max-ltd-c-166.htm : nike air max ltd
    http://www.shoesbuying.com/nike-air-max-tn-c-167.htm : nike air max tn
    http://www.shoesbuying.com/nike-rift-shoes-c-162.htm : Nike air rift
    http://www.shoesbuying.com/nike-yeezy-c-184.htm : Nike air Yeezy
    http://www.shoesbuying.com/nike-airmax-c-154.htm : nike airmax
    http://www.shoesbuying.com/nike-airmax-c-155.htm : Nike air max 90
    http://www.shoesbuying.com/nike-airmax-c-157.htm : Nike air max 97
    http://www.shoesbuying.com/nike-birds-nest-shoes-c-152.htm : nike birds nest shoes
    http://www.shoesbuying.com/nike-dunk-high-c-177.htm : nike dunk
    http://www.shoesbuying.com/nike-shoes-c-168.htm : nike RT1 shoes
    http://www.shoesbuying.com/nike-sb-c-180.htm : nike SB
    http://www.shoesbuying.com/nike-shox-shoes-c-156.htm : nike shox shoes
    http://www.shoesbuying.com/nike-shox-shoes-c-158.htm : Nike shox OZ shoes
    http://www.shoesbuying.com/nike-shox-shoes-c-159.htm : Nike shox R2 shoes
    http://www.shoesbuying.com/nike-shox-shoes-c-160.htm : Nike shox R3 shoes
    http://www.shoesbuying.com/nike-shox-shoes-c-163.htm : Nike shox R4 shoes
    http://www.shoesbuying.com/nike-shox-shoes-c-164.htm : Nike shox R5 shoes
    http://www.shoesbuying.com/nike-shox-c-171.htm : Nike shox TL3
    http://www.shoesbuying.com/nike-trainers-lovers-c-118.htm : nike trainers lovers

    http://www.racketsoffer.com/ : tennis rackets
    http://www.racketsoffer.com/wilson-racquets-c-19/ : Wilson tennis rackets
    http://www.racketsoffer.com/head-racquets-c-18/ : HEAD tennis rackets
    http://www.racketsoffer.com/babolat-racquets-c-17/ : Babolat tennis rackets

  10. Rewrite the financial system and the ERP system. It will probably take less time, and you'll have a more stable, robust, featureful, user friendly, and testable system to boot.

    I hate those "enterprise" apps. I'm surprised though that you're even talking about using SQL, since things like Siebel, Peoplesoft, etc. do their worst to keep your hands off of your data in your database.

    Seriously, though, that's the biggest challenge. Documenting your acceptance criteria and pointing tests to them is about all you can do. I'm working (a few hours a week) on a fitnesse-like tool, but there isn't a really good way to do it.

    Cucumber (cukes.info) is popular, but it runs into the same problem — it's code, and a very limited grammar. I'm convinced that plain english documentation tied to good old fashioned test scripts (either manual or automated) is the way to go.

    I don't want to say it, but something like Test Director/Quality Center is where I'm headed, but using open source tools like Selenium and xUnit.

Leave a Reply

Your email address will not be published.