Cloudy with a Chance of ATRT

Our experience at IDT shows that using virtual and cloud test environments combined with automated software testing using Automated Test and ReTest (ATRT) solutions can help reduce testing infrastructure costs. Testing areas affected may include interoperability, system testing, functional testing, component testing and integration testing, among others. Additional benefits of combining automated testing in a virtualized cloud environment include a more reliable system, improved testing quality, and reduced test effort and schedule.

What do ATRT solutions do?

ATRT solutions in a virtualized cloud environment can support a stable, scalable automated testing infrastructure that extends across one or many server farms, across one or many system(s) under test (SUTs) and works with a common set of cloud computing concepts to support a broad virtualized enterprise automated test environment.   This specific testing setup allows the use of virtualization in a specialized way to reduce the need for purchasing, storing and maintaining various expensive Test Environment (TE) hardware and software. Proper virtualization setup provides a multi-user access automated testing solution that allows users to implement and reuse ATRT, along with all testing artifacts, on a provisioning basis. Additionally all related automated testing activities and processes, i.e. test case and requirements import; requirements traceability, automated test creation and execution, and defect tracking take place in this virtualized cloud environment.

Combining ATRT solution test efficiency with the hardware cost savings gained from implementing in a virtualized/cloud environment results in significant savings.   For example 20 Virtual Machines (VMs) fit on one server in our virtual environment – allowing for a cost savings of 20 to 1– in the test environment. Additionally, in the virtual environment, the SUT VM can be located anywhere on a connected network and doesn’t need to be located physically in the same VM as the testing VM.

Why are ATRT solutions uniquely qualified to be deployed in a virtualized cloud environment?

In order to support a virtualized automated test environment, it is critical the automated testing solution itself is scalable.  For example, the ATRT technology allows for N number of concurrent tests to run or N number of serial tests, depending on the test type required. All of the tests and test outcomes are stored in the ATRT: Test Manager repository for example, for access by any subscriber (or user) of the ATRT virtualized environment. A subscriber/user can be a developer or tester or anyone on the program with ATRT: Test Manager user access privileges.

This virtualized test environment example setup supports live migration of machines; load balancing; easy movement of machines to different servers without network interruption and allows any upgraded VM to run on any server.  As a result, it is also important that in a virtualized environment an automated testing solution is not only scalable but that its artifacts are portable.  ATRT: Test Manager for example can test systems independent of OS or platform so it is able to support applications running on both Windows and Linux, providing flexibility to migrate machines without the constraint of the OS or platform the automated test tool can support.

Additionally, an automated testing solution should be selected that does not need to be installed on the SUT. ATRT: Test Manager is an example of a solution that does not need to be installed on the SUT and instead is communicating with the SUT via a VNC Server or the RDP protocol which transmits the necessary information back to the ATRT: Test Manager for specific test actions. Few automated software testing solutions exist that don’t need to be installed on the SUT. The typical automated testing tool needs to be installed on the System Under Test (SUT) so it can link to the GUI coding libraries to get the object properties of the GUI widgets and/or pull information out of the Operating System’s window manager for example, in order to create an automated test baseline. Installing an automated testing tool on the SUT however is generally not desired, because 1) the installation modifies the system configuration (the testing system environment should be identical to the production system environment) and 2) it doesn’t lend itself to cloud computing because of the additional tool installations on each SUT.

In our proposed virtualized test environment (VTE) the SUT VM can be located anywhere on a connected network and doesn’t need to be located physically in the same VM as the ATRT VM. This allows for tremendous flexibility. For example, multiple ATRT VMs can run in the VTE connecting to hundreds of SUT VMs. However, in the typical automated testing setup where the tool needs to be installed on the same machine as the SUT, a 1 : 1 setup is required, i.e. one automated testing tool for each one SUT, negating some of the savings expected in a VTE.

For a more detailed discussion of our virtualization efficiencies using ATRT solutions, please see http://www.crosstalkonline.org/storage/issue-archives/2013/201307/201307-Dustin.pdf