Continuing our discussion of metrics that can be applied to testing processes, Automation Progress is another useful metric. It refers to the percent of automatable test cases and asks the question, “How many test cases have been automated at a given time?” Basically, this metric evaluates how well one is doing in the goal of automated software testing. The goal is to automate 100% of the “automatable” test cases. This metric is useful during the various stages of automated testing development.
AA # of actual test cases automated
AP (%) = ————- = ( ——————————————————)
ATC # of test cases automatable
- AP = Automation Progress
- AA = # of actual test cases automated
- ATC = # of test cases automatable
The Automation Progress metric is a metric typically tracked over time. In the case below, time is being measured in “weeks”.
A common metric closely associated with progress of automation, yet not exclusive to automation is Test Progress. Test progress can simply be defined as the number of test cases attempted (or completed) over time.
TC # of test cases (attempted or complete
TP = ———– = ( ————————————————————– )
T time (days/weeks/months, etc)
- TP = Test Progress
- TC = # of test cases (either attempted or completed)
- T = some unit of time (days / weeks / months, etc)
The purpose of this metric is to track test progress and compare it to the plan. This metric can be used to show where testing is tracking against the overall project plan. Test Progress over the period of time of a project usually follows an “S” shape. This typical “S” shape usually mirrors the testing activity during the project lifecycle. Little initial testing, followed by an increased amount of testing through the various development phases, into quality assurance, prior to release or delivery.
This is a metric to show progress over time. A more detailed analysis is needed to determine pass/fail, which can be represented in other metrics. See other blogs under “Metrics” for additional information.
Some information taken from: Dustin, Elfriede, Thom Garrett, and Bernie Gauf. Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality. Upper Saddle River, NJ: Addison-Wesley, 2009. This book was authored by three IDT employees and is a comprehensive resource on AST.