www.ebbemunk.dkarrowThe Economics of Software Testing

The Economics of Software Testing

Speech on the Eurostar 2000 Conference, December 2000, Copenhagen, by Les Hatton, Oakwood Computing, Surrey, U. K., and the Computing Laboratory, University of Kent. mailto:lesh@oakcomp.co.uk. Minutes by Ebbe Munk.

Risk and Uncertainty

It is fundamentally impossible to quantify risk

People want to take risks

Everybody has a propensity to take risk

Conclusions about risk

  1. Everyone else is seeking to manage risk too
  2. Everybody is guessing. If they knew, it's not risk.
  3. Guesses are extremely influenced by beliefs
  4. The behaviour of others and the behaviour of nature are your risk environment
  5. Unless people’s propensity to take risk is reduced:
  1. Science will continue to invent new risks
  2. In the dance of the risk thermostats, the music never stops

(Source: The risk thermostat, J. Adams 1984)

Examples and Sources of Risk and Failure

Increased risks by large usage in embedded control systems:

J. Adams found in 1984, that 1/3 of all faults only failed less than once every 5000 execution years. In an embedded control system in a car with say 1,000,000 copies around the world, every such error will appear in one car every four days.

Web sites, which are heavily used, exhibit exactly the same kind of behavior. There is clear evidence with both embedded control systems and web development that the increased risks produced by unusually large usage are being ignored.

Computer Languages are not improved over Time

Language standardisation disobeys control process feedback in several important ways:

Programming languages are designed by individualists. Control process feedback is designed by the hierarchy.

The problem is that we are trying to use a hierarchy technique on an individualist technology.

Conclusion: In general, reduced risk is compensated by adding more features. We must reduce the propensity of software managers and engineers to take risks:

A prediction: Improvements in software testing will not in general lead to improved reliability. They will simply lead to more features.