Detecting faults vs. Exposing failures: Orthogonal measures of test suite effectiveness

One of the most important decisions we make in software testing is the choice of test data; and one of the most important factors that drives and determines this choice is the way we measure and compare test suite effectiveness. Most measures of test suite effectiveness in use nowadays equate test suite effectiveness with the ability to detect program faults. We argue that there is an alternative way to measure the effectiveness of a test suite, which is to equate its effectiveness by its ability to expose program failures. This raises three immediate questions: First, are these metrics identical? If not, are they statistically correlated? If not, which is a better measure of test suite effectiveness? In this paper, we discuss these questions on the basis of analytical and empirical arguments.