Often when people think of automated software testing they think of simply automating the actions a user or test engineer takes to interact with the system’s GUI and verify the system’s response. For many systems and tests, automated GUI-based testing is a great strategy. However, GUI-based testing has limitations and may not be the best strategy. Message-based testing is a great complement to use in conjunction with GUI-based testing, and in some cases it may just be a better strategy all together.
For example, in some cases attempting to verify the system response is correct is not possible from the GUI or may not be practical. Complex GUI’s with non-numeric information that is constantly being updated is one example that comes to mind. However, evaluating the data in the messages associated with the information being displayed can be a very effective approach to evaluating the system response.
Message-based testing is also hugely beneficial in situations where multiple entities are responsible for developing components of a system. From a developer’s perspective, the IDS (Interface Design Specification) is never absolutely comprehensive. There are always parts that are open to interpretation. This interpretation by different entities can lead to problems that don’t get discovered until the components of the system are assembled and then they are very expensive to change.
Message-based testing of a system at an early stage can prevent these issues by testing how the system performs when it receives messages that both conform to the specification and do not conform. The system can be designed to cope with messages that don’t match up with what is expected, and to perform gracefully in the event of receiving something that is not quite right. Systems are usually designed to handle poorly formatted messages. However, messaging tests provide great utility for finding issues with the system’s performance when it receives a sequence of correctly formatted messages in the incorrect order.
In the DoD world, where many of these messages are transmitted between subsystems and other systems, we use automation to send messages in any order, containing any data, to bring bugs to the surface. Developing a message-based test allows for virtually unlimited combinations of messages to be exchanged with the system, providing a thorough shakedown before the specific system being tested has to be integrated into the full suite of systems.