Case Study: Automated Endurance Tests
In this case study, we discuss IDT’s success in applying automation to a very common type of software testing called an “endurance” or “longevity” test. In general, endurance or longevity tests, if done manually, are very time consuming and labor intensive. The environment we were faced with involved a very large, complex, mission-critical system comprised of hundreds of applications and millions of lines of software. This specific endurance test was designed to execute for 25 hours and required a team of seven people to conduct manually.
The automated test strategy was to apply ATRT (Automated Test and ReTest) to the endurance test in order to reduce the number of people required to conduct the test. For this specific test, the system requirements being verified dictated that the execution time of the test not to be changed from the original design of 25 hours. During the strategy phase, the operator actions which could be automated and would yield the greatest reduction in manpower were identified and prioritized for implementation.
To meet the goals for automated software testing for this case, IDT utilized the ATRT: Test Manager. The first step was to develop automated test procedures for sections of the test identified as the highest payoff in the strategy. The overall endurance test included stress and regression testing sections. The stress test puts a heavy load on the system trying to exhaust the various system resources. The regression tests were broken into three separate groups of test. The first regression test focused on software components related to interfaces between the system. The second regression test dealt with software components related to communications, control processing and reporting capabilities. The third regression test verified software components dealing with threat identification and evaluation, logic for selecting actions, and implementation processing.
Using ATRT: Test Manager to assist operators in conducting the 25 hour endurance test enabled a reduction in staffing from seven operators to two. In addition to meeting the customer’s primary objective of reducing the number of people required to run this very lengthy test, some additional benefits have been: (1) the results of the test are now documented in a consistent and thorough manner; (2) in the event of a failure during the test critical system, information associated with the system state is now available through ATRT: Test Manager to support an understanding of what happened; and (3) the conduct of the test is more consistent and able to be run more often because scheduling the event requires significantly fewer people.