ROI Calculations for Automated Software Testing
Calculating the ROI for the implementation of an Automated Software Testing program is a detailed, worthwhile activity that includes considering various criteria. An experienced partner like IDT can walk you through various metrics and factors that should be considered. These usually include: test setup time savings, test development time savings, test execution time savings and test evaluation/diagnostics time savings. Here is a bird’s eye view of our approach to ROI evaluation.
Test Environment Setup Time Savings
First, it is important to compare the time estimates for manual versus automated test environment setup tasks. Various test environment preparation activities, as listed below, can be automated.
- Test environment setup
- Component installation
- Re-baselining a test environment
- Test data population
- Configuration verification
Test Development Time Savings
To estimate the savings that automation lends to test development time it is important to consider both the number of tests planned and the estimated time to develop one test. If the exact number of tests is not known, an estimate based on the number of requirements can be used. The estimated time to develop one test is based on the average time it takes to develop one test in minutes. Actual numbers from previous manual tests are very helpful. Initially, we estimate a 50% increase in the time it takes to develop automated tests. The total test development time is approximately the number of planned tests times the estimated time to develop one test—both manually and with automation.
Test Execution Time Savings
- Test execution time savings is examined based on the following criteria:
- Number of tests planned
- Estimated time to execute each test (minutes)
- Estimated number of times the test will be rerun (including future builds)
Multiplying these three factors times each other gives the total test execution time—both manually and with automation. It is important to keep in mind that with automated testing, tests can be executed at any time without tester involvement, i.e. when you go home, to a meeting, over the weekend, etc., whereas with manual tests that is not the case.
Test Evaluation/Diagnostics Time Savings
Comparing test outcomes to expected results is an important step in the test evaluation process. It is beneficial to automate these diagnostics. The number of test outputs to diagnose is an estimate of the total number of results needing evaluation. This number can be multiplied by the estimated time to evaluate each test in minutes. Then these can be multiplied by the number of test iterations. Again, basing these numbers on records from previous manual testing is helpful. It is usually about 10 times faster to evaluate tests with automation when compared to manual testing. A side by side comparison of these diagnostics—both automated and manual—reveals substantial time savings with automation.
Once approximate numbers are ascertained for these four areas, they can be charted and compared. Contact IDT to discuss a full ROI evaluation of implementing automation for your system.
Some information taken from: Dustin, Elfriede, Thom Garrett, and Bernie Gauf. Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality. Upper Saddle River, NJ: Addison-Wesley, 2009. This book was authored by three current IDT employees and is a comprehensive resource on AST. Blog content may also reflect interviews with the authors.