Management Toolkit

Software testing should be a risky business

Rupert Goodwins ZDNet.co.uk

Published: 01 Jul 2005 11:45 BST

Acutest is an independent testing consultancy which works primarily with blue-chip clients, and claims that 50 percent of testing done on technology-centred change projects is wasted.

ZDNet UK spoke with the company's chief executive Barry Varley about the role of testing in software projects.

Q: You've conducted surveys of IT professionals and their approach to testing. What have you found?
A: There's a huge variability in the answers they give, and a massive discrepancy between what they think they've done and how well they have done. In one simple exercise, they thought they'd scored around 60 percent when the actual result was 16 percent. Another area where there's a great difference between perceptions and reality is in timing. When we asked how long do you think a particular test would take to do in real life there was a massive underestimate — on average, by 50 percent. Everyone is surprised — everyone thinks it's much simpler than it is.

So why didn't people learn from the Y2K experience, when testing became a major focus for everyone?
With Y2K, everyone thought it was a one-off. It was approached as a special case, something that would never have to happen again. Nobody applies the lessons learned.

So why do you think so much testing is wasted?
We looked at a lot of test areas and put them on a graph where the impact of failure of a particular area was plotted against the likelihood of it happening. The vast majority of testing was concentrated in areas that were unlikely to fail and wouldn't have much of an impact if they did. People don't look at testing in a risk-based way — they need to do far more risk analysis. Instead, they do things like test in order of component delivery, something that needs little planning but doesn't reflect the relative importance of each component.

Where are the major failures in methodology?
People don't involve the business in testing, and don't involve testing in the design. They're also oblivious to ambiguities in the specification, especially when it comes to offshoring. Context is lost. People say "when it comes back, we'll test it", but what you get back is not what you were expected. Also, a whole bunch of business changes are left too late, when they should be integrated into the design and testing as soon as possible.

Do you see things getting better?
There are changes. Governance demands hard information during a project, which testing is a part of. Businesses are seeing good testing and better involvement of the business as a differentiator, doing more business process testing, which has never been done before. And by looking at points of failure, you can identify risk areas. Nobody's got time to test everything, so have to do risk analysis, and that means you can concentrate not just testing but design and implementation effort where it does most good.