You Get What You Inspect

Automated Software Testing Gears

As Americans, we love to espouse the virtues of a quality product. When I was growing up in the Midwest, my father was a member of ASQC—the American Society of Quality Control. This was an entity devoted to ensuring that all things quality permeated the manufacturing industry. Much like we at IDT are striving to ensure that Automated Software Testing permeates all sectors of the software industry. Anyways, he would travel to exotic, far off places like Ohio to take rigorous ASQC courses in statistics. Then, with only a caliper, a micrometer and the wonderful ability to tell vendors their parts were sub-par, my father would ensure that only the best products he inspected would be allowed on his Kiekhaefer outboard motors (later Mercury and then Brunswick).  “You get what you inspect,” he would say, and to this day, Mercs are still the best outboard motors in the world (in my opinion).

I would ask him if he tested everything and dad just said, “No, I use statistics.”

Statistics: How Much is Enough?

Anyone who has taken a course in statistics probably knows that an appropriate sample size is critical for a good sample…too big and you waste resources, too small and you miss a more representative sample.[1] The same goes for stress testing…stress the product too little and you don’t know where it will fail, stress it too much and you end up ruining the product.

Fast forward about 10 years, and his son is an officer in the US Navy supervising stress tests on cables, lines, chains, wire ropes and anything that would hold a load of critical importance. We would constantly test and re-test these components to ensure quality. Our test load would be exactly two times the anticipated normal load[2] (why two times and not pi or natural e, I’m still not sure). Nonetheless, we were pretty much guaranteed that the thing wouldn’t break if we used it normally. The key idea was testing and re-testing. As components wear out or new components were introduced into the mix, we tested and re-tested to ensure quality.

The New Stress Tests: Automated Software Testing

Just as in the sample and load testing examples above, software likewise needs to be rigorously tested. Too little testing and products suffer; too much testing or testing in the wrong area and resources can be wasted (although I still think one can NEVER do too much testing!). Software COCOMO[3] models indicate anywhere from 30[4] to 40 percent of the cost of a software project is tied up in testing. Sustaining software via testing and re-verification of requirements can account for 60 to 90 percent of the total life cycle costs![5] Clearly, automating these processes will reduce the total overall cost as well as improve the product. You get what you inspect.

This blog was written by Tim Schauer, IDT Senior Systems Analyst.


[3] Boehm, B.W. Software Engineering Economics Englewood Cliffs, NJ. Prentice- Hall, Inc., 1981.

[5] http://www.stsc.hill.af.mil/resources/tech_docs/gsam4/chap16.pdf