Another automated software testing metric we want to consider is Percent of Automated Testing Test Coverage. That is a long title for a metric to determine how much test coverage is the automated testing actually achieving? It is a metric which indicates the completeness of the testing. This metric is less focused on measuring how much automation is being executed, but rather on how much of the product’s functionality is being covered. For example, 2000 test cases executing the same or similar data paths may take a lot of time and effort to execute, but this does not necessarily equate to a large percentage of test coverage. Percent of automatable testing coverage does not specify anything about the effectiveness of the testing taking place, it is a metric to measure its’ dimension.
AC automation coverage
PTC(%) = ———— = ( ———————————– )
C total coverage
- PTC = Percent of Automatable testing coverage
- AC = Automation coverage
- C = Total Coverage (KLOC, FP, etc)
Size of system is usually counted as lines of code (KLOC) or function points (FP). KLOC is a common method of sizing a system; however, FP has also gained acceptance. Some argue that FPs can be used to size software applications more accurately. Function Point Analysis was developed in an attempt to overcome difficulties associated with KLOC (or just LOC) sizing. Function Points measure software size by quantifying the functionality provided to the user based logical design and functional specifications. There is a wealth of material available regarding the sizing or coverage of systems.
The Percent Automated Test Coverage metric can be used in conjunction with the standard software testing metric called Test Coverage.
TTP total # of TP
TC(%) = ———— = ( ————————————— )
TTR total # of Test Requirements
- TC = Percent of Testing Coverage
- TTP = Total # of Test Procedures developed
- TTR = Total # of defined Test Requirements
This measurement of test coverage divides the total number of test procedures developed, by the total number of defined test requirements. This metric provides the test team with a barometer to gauge the depth of test coverage. The depth of test coverage is usually based on the defined acceptance criteria. When testing a mission critical system, such as operational medical systems, the test coverage indicator would need to be high relative to the depth of test coverage for non-mission critical systems. The depth of test coverage for a commercial software product that will be used by millions of end users may also be high relative to a government information system with a couple of hundred end users.
Some information taken from: Dustin, Elfriede, Thom Garrett, and Bernie Gauf. Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality. Upper Saddle River, NJ: Addison-Wesley, 2009. This book was authored by three current IDT employees and is a comprehensive resource on AST.