Much has been reported in the news lately about several high-visibility software systems– e.g. commonapp.org, healthcare.gov, united.com, etc.– whose extensive defect leakage affects many. We calculate defect leakage in software systems as follows: the number of high priority defects found in the production phase, divided by the number of high priority defects found in the testing phase, multiplied by 100.
Is your defect leakage out of control, also?
We credit automated software testing – among other factors, such as Continuous Integration (CI) – for our <0.01% defect leakage record for solutions we test and release. Applying automated software testing in a CI environment to our solutions, along with applying automated testing on numerous other programs, we have been able to show that it 1) is more efficient than manual testing, 2) increases testing coverage, and 3) improves software quality. In this blog, we will use graphs to illustrate the impact of this additional testing on defect leakage, and its resulting improved software quality.
Software reliability models predict the results from continued testing and overall defect statistics associated with the system under test. See Figure Example Software Model Prediction which shows that generally more defects are uncovered the longer the allocated test time.
We can use software reliability models to forecast the impact of additional testing on software quality.
Figure Example Impact of Additional Testing provides examples of how many more defects can be found and resolved with this type of “additional” testing.
Since Automated Software Testing is faster and we can test more with less, plus it allows for increased test coverage we can therefore infer that automated software testing uncovers more defects and thus yields Higher Quality Software. See Figure Automated Testing is an Enabler to improved Software Quality.
Automates software testing, among other factors, is an enabler to software quality and will help you keep your defect leakage in check. For a discussion on other automated testing metrics, please take a look here.