Background:

System of system testing is complex, costly, and time consuming.  These factors are driven primarily by the need to not only assess the performance of each individual system but also assess the aggregate performance of the system of systems.  Performance assessments typically require the conduct of tests, real-time evaluation of results and also significant post-test analysis.

Strategy:

Based on a review of the customer’s current manual approach, Innovative Defense Technologies (IDT) developed an automated test strategy using Automated Test and ReTest (ATRT).  ATRT is a process, development methodology and software suite used to apply automated software testing and data analysis to large complex systems.  The overall objective was to design, develop, and deliver an automated test and re-test capability which would enable a substantial reduction in the cost, time, and resources associated with the current test and evaluation methods, while also increasing the quality and breadth of the testing conducted. An analysis of the current test and analysis approach showed that a significant amount of time and effort was required in the post test analysis phase–specifically the analysis associated with Event Reconstruction.  As result, the focus for this project was to provide an automated event reconstruction capability in support of system of system testing.

Approach:

To meet program objectives IDT leveraged the current ATRT: Analysis Manager (ATRT: AM)  with added enhancements in the areas of Data Management and Reporting/Presentation of Metrics.  In addition, capability was added to the ATRT: AM to support the automated association of truth data with the track data/output data produced by each individual system. The delivered solution with ATRT: AM provided an automated event reconstruction capability in support of system of systems post-test analysis.

Conclusion:

System of system testing is necessary to verify systems are both interoperable and meeting overall performance objectives.  With multiple systems involved, each of which has generally been developed by different organizations, testing at the system of systems level is challenging. Among the many challenges is event reconstruction phase of the post-test analysis.

The automated event reconstruction capability delivered with the ATRT: AM was able to provide a 95% improvement (in time and manpower).  In addition to a substantial reduction in time and manpower the automated event reconstruction capability also provides a consistent and repeatable methodology resulting in a more accurate assessment of the system’s performance.