In the previous blog on metrics dated August 29,  we mentioned that one of the characteristics of a good metric is simplicity.  Albert Einstein once said,

“Make everything simple as possible, but not simpler.” 

When applying this wisdom towards automated software testing, you will see that:

  1. Simple reduces errors
  2. Simple is more effective
  3. Simple is elegant
  4. Simple brings focus

It is important to generate a metric that calculates the value of automation, especially if this is the first time the project has used an automated software testing approach. The test team will need to measure the time spent on developing and executing test scripts against the results that the scripts produced. For example, the test team could compare the number of hours to develop and execute test procedures by the number of defects documented that would not likely have been revealed during a manual test effort.

Sometimes it is hard to quantify or measure the automation benefits. For example, often defects are discovered using automated testing tools, which manual test execution could not have discovered. For example, during stress testing 1000 virtual users execute a specific functionality and the system crashes. It would be very difficult to discover this problem manually, using 1000 test engineers. Another way to minimize the test effort involves the use of an automated test tool for data entry or record setup. The metric, which applies in this case, measures the time required to manually set up the needed records versus the time required to set up the records using an automated tool.

Consider the test effort associated with the system requirement that reads, “The system shall allow the addition of 10,000 new accounts”. Imagine having to manually enter 10,000 accounts into a system, in order to test this requirement! An automated test script can easily support this requirement by reading account information from a file through the use of a looping construct. The data file can easily generated using a data generator. The effort to verify this system requirement using test automation requires far fewer number of man hours than performing such a test using manual test methods.

Automated software testing metrics can be used to determine additional test data combinations. For example, with manual testing you might have been able to test ‘x’ number of test data combinations; with automated testing you are now able to test ’x+y’ test data combinations. Defects that were uncovered in the set of ‘y’ combinations are the defects that manual testing may have never uncovered . If you can quantify something, then you can explain it in more detail and know something more about it. If you can explain it, then you have a better chance to attempt to improve upon it.  Metrics are essential in your overall automated testing strategy.