Case Study: Automated Regression Testing of a Shipboard Control System
Regression testing is a type of software testing that involves retesting software after changes or enhancements have been introduced to a system. As a result, one of the goals of regression testing is to assess whether functionality previously delivered continues to operate as expected following these changes. The most common technique used to meet regression testing goals is through assessing requirements previously verified. Automated regression testing applies an automated test strategy to the regression testing effort.
In this case study, we will discuss IDT’s success in applying automation to the regression tests of shipboard control systems. Automated regression tests were applied to the highly-complex control systems located on ships. These control systems monitor thousands of signals associated with numerous shipboard systems such as pump valves, fuel tank levels and valves, alarm management systems, fire suppression systems, ballast systems, potable water systems and human waste containment systems. Because of the size of the software systems, traditional regression testing of these systems took about eight weeks to perform and only covered about 10% of the associated signals.
IDT’s automated test strategy was to apply Automated Test and ReTest (ATRT) technology to support the project’s need to expand their regression test coverage from approximately 10% to more than 90% while at the same time not increasing the time allocated for regression testing.
The current manual regression tests were conducted by stimulating the software by injecting signals into the system and then observing the operator screens to check for status and alarms. For example, a change would be sent to the pressure gauge by lowering the pressure below a certain threshold. Then actual output values would be observed on the operator console to determine if the software had correctly detected the change and sent the necessary “low pressure” alarm. After reviewing the nature of the manual tests, IDT determined that the application of ATRT: Test Manager would facilitate meeting the project’s objectives. ATRT: Test Manager is designed to support large, complex systems of this type and provides the capability to automate operator inputs/actions and verify the response.
Using ATRT: Test Manager in the automation of shipboard control system regression testing significantly increased the overall test coverage while it simultaneously maintained the currently allocated time for regression testing. By applying automation, the test team was able to regression test 100% of signals within the same eight week time period, whereas previously only about 10% of signals were able to be tested. The test team achieved a tenfold productivity improvement (10% to 100%) using automated testing.