NAVAIR – Public Release 2014-953 Distribution Statement A: “Approved for public release; distribution is unlimited.”
Background:
The US Naval Air Systems Command (NAVAIR) supports naval aviation aircraft, weapons, and systems including their design, development, test, and sustainment. NAVAIR’s program office PMA-281’s responsibilities include both mission planning systems and command and control systems. These complex software systems must meet thousands of requirements, support multiple platform configurations, and interface to multiple other subsystems. In addition, the systems are designed, developed, and tested by multiple contractors and government organizations that are geographically dispersed. Because of the mission critical nature of the systems, software changes require retesting of requirements across multiple configurations to ensure the systems are suitable for deployment.
NAVAIR PMA-281’s interest was to be able to more rapidly field new capabilities while also increasing the quality of the software they delivered without increasing the cost of their system.
Strategy:
Innovative Defense Technologies (IDT) worked with NAVAIR PMA-281 to develop a strategy that would support NAVAIR’s objectives. Analysis of the current NAVAIR processes showed that employment of IDT’s patented Automated Test and ReTest (ATRT) methodology and technology would yield significant benefit. The strategy consisted of the following components:
- Automate execution of tests. The time required to manually conduct retesting of software changes was limiting the pace at which NAVAIR could deliver the system; but, once automated, retesting could be completed in significantly less time and could run unattended.
- Test earlier and expand the number of requirements, permutations, and configurations tested. Manpower and time constraints were limiting the number of requirements, permutations, and configurations that could be tested with each software release. As a result, there was a higher risk the system could be delivered with unknown defects. In addition, the majority of the testing was conducted near the end of the delivery cycle, when identified software problems are most expensive to fix. Automation with ATRT would enable more requirements, permutations, and configurations to be tested, which would reduce the risk of latent software defects. Additionally, since the tests would be automated, it would enable the testing to be conducted earlier in the delivery cycle.
- Reuse test cases and share test results among the contractor and government teams responsible for the system. Each organization was maintaining its own tests with limited capability to share detailed results and/or re-create observed problems.Time was being spent developing, modifying, and re-running the same or very similar tests. In addition, when a problem was identified there was often limited information available to re-create the problem so that it could be fixed. ATRT would provide a means for distributed teams of engineers to efficiently create, share, and reuse tests and also enable the capability to share test results, including the step-by-step reconstruction necessary to re-create a problem when identified.
Results:
NAVAIR PMA-281 has successfully employed their strategy with ATRT and realized the following results:
- Increased testing efficiency by greater than 75%. The result is significantly less time and manpower is required to conduct testing.
- The number of requirements, permutations and configurations being tested has increased along with consistency of the testing. In addition, test teams have been able to identify software defects earlier in the schedule.
- Automated test cases are being shared and reused across the responsible contractor and government teams. Besides the efficiency of reusing test cases, the time and scope of incorporating automation is also being accelerated.
Download a PDF of IDT’s NAVAIR Case Study.