Automated Software Testing: From Problems to Solutions
As discussed in the July 15th blog, there are three basic categories of system testing: white-box, black-box and gray-box. All three types are required to thoroughly test a system. And all test programs are dependent upon the development of high-quality tests.
As testing is conducted, there are various kinds of problems that can be encountered, but the top four are as follows:
- The test reveals an error in the system and the operation is aborted.
- Test results are incorrect or differ from the expected outcome.
- A false positive is issued by the test; a component fails but no error is reported.
- A false negative is issued by the test; there is no failure, but an error is reported.
To prevent some of these inaccuracies, and to more easily resolve some of the issues tests reveal, it is important that tests are designed to generate specific error messages. Tests should also produce a detailed error log, which can give the tester more information. These results will enable the tester to evaluate the system more precisely.
Some types of testing still require human interaction, especially usability testing. This type of manual testing measures the “user-friendliness” of a system. Web page developers are particularly beholden to this type of testing, as usability is essential to the success of their business.
While manual testing is needed to certify usability, for many types of tests, manual testing is not an efficient choice. Inconsistent inputs are common with manual testing and hence it is error-prone. Manual testing can be very time-consuming, cumbersome and limited. Aside from usability tests, many other tests are much better suited to automation.
Automated Software Testing is the application and implementation of software testing technology throughout the entire software testing lifecycle, with the goal of improving testing efficiency and effectiveness. Automated testing provides many important benefits, which will be discussed at length in future blogs. It is highly efficient for tests that require repeatability, bottom-up testing, and code analysis. There are many system tests that are well-suited for automation, including:
- Regression tests
- Functional tests
- Interface tests
- Concurrency tests
- Performance tests
- Endurance tests
- Security tests
- Stress tests
- Multi-component thread tests
Automated software testing will almost always be more efficient than manual testing for the types of tests listed above. We will discuss how to assess what to automate in an upcoming blog. See the IDT website, or visit again soon for more in-depth information.
Some information taken from: Dustin, Elfriede, Thom Garrett, and Bernie Gauf. Implementing Automated Software Testing: How to save Time and Lower Costs While Raising Quality. Upper Saddle River, NJ: Addison-Wesley, 2009. This book was authored by three current IDT employees and is a comprehensive resource on AST. Blog content may also reflect interviews with the authors.