Case Study: Automating Performance Tests for Complex Systems
This case study will focus on applying automated software testing to performance and functional tests against complex system of systems. These tests are run on multiple systems with multiple operator views. These operators interact with the various consoles and are required to coordinate actions that appear on multiple display consoles throughout the test. They also need to verify application performance response time, measure screen render time, and verify messages between applications and between sub-systems. The main focus or goal of this particular set of tests was to be able to ensure the tests can be conducted in consistent and reliable manner while also reducing the test execution time. Currently, one run of the manual test takes approximately 35 hours to complete.
Innovative Defense Technologies’ automated test strategy was to focus on reducing the amount of time required to complete the performance / functional tests. ATRT: Test Manager was selected because it lends itself well to running test cases unattended, on complex systems, with the ability to interact and verify information from multiple display consoles simultaneous with the measurement and verification of network or message traffic. Since ATRT: Test Manager is non-intrusive to the system under test it is also ideal for performance tests such as described in this case study. To obtain accurate results in performance tests, non-intrusiveness is particularly important as not to introduce a load on the system under test.
After a review of the detailed test requirements, ATRT: Test Manager test cases were then developed. The test cases were able to be executed automatically, running through all the performance and functional tests independent of any operator interactions. ATRT: Test Manager was able to accurately measure and report on the vital performance measures such as display rendering time, message verification, and interaction between sub-systems and other components.
Overall time to complete the desired performance and functional tests was reduced by 52% relative to historical manual test approach. In addition, the performance and functional tests are able to be repeated consistently and reliably using ATRT: Test Manager. Finally, a series of detailed reports of test results is now also immediately available at the completion of each test.