Empirical validation in software testing increasingly relies on mutants, motivated by reported strong correlations between mutant scores and real fault detection. This study demonstrates that those correlations are largely an artifact of test‑suite size, a confounding factor. Using two large datasets—CoREBench (C programs) and Defects4J (Java programs)—the authors show that when test‑suite size is controlled, the correlation between mutation score and real fault detection becomes weak (≈0.05–0.20). Nevertheless, test suites that achieve the highest mutation scores do detect significantly more faults than randomly selected suites of the same size, indicating that mutants provide useful guidance for improving test suites even though they are poor proxies for fault‑detection effectiveness.
Key Points
The strong correlation between mutation score and fault detection reported in earlier studies is primarily due to the confounding effect of test‑suite size.
When controlling for size, all correlations become weak (Pearson/Kendall ≈0.05–0.20), regardless of programming language (C or Java).
Both mutation score and test‑suite size significantly influence fault detection, but neither alone (nor their combination) predicts fault detection well (pseudo‑R² values remain moderate).
Despite weak correlations, test suites with the highest mutation scores (top 10%) yield statistically significant fault‑detection improvements for ≈60% of Defects4J faults and ≈87% of CoREBench faults.
Fewer than 1% of mutants represent real fault behaviour; the majority of mutants are “irrelevant,” introducing noise into mutation scores.
The study resolves contradictions in the literature by accounting for test‑suite size and by evaluating both correlation and fault‑detection-rate metrics.
Concepts
Mutation analysis: The process of introducing small syntactic changes (mutants) into a program. A test case kills a mutant if it causes the program to produce a different output; otherwise the mutant is live. The mutation score is the ratio of killed to total non‑equivalent mutants.
Real faults: Naturally occurring defects that have been isolated from version histories (e.g., from Defects4J or CoREBench).
Confounding factor: A variable (here, test‑suite size) that influences both the independent variable (mutation score) and the dependent variable (fault detection), producing a spurious correlation when not controlled.
Correlation method: Computes the correlation coefficient between mutation scores (or other criteria scores) and fault‑detection ratios across a set of test suites.
Fault detection method: Measures the actual fault‑detection rate of test suites that achieve a predefined score level (e.g., 100% mutation score) and compares it to a random baseline.
Details
Study Design and Subjects
The experiment used two collections of real faults:
CoREBench: 70 fault instances from four C programs (Coreutils, Findutils, Grep, Make). Two faults (identifiers 57 and 58 from Make) were excluded due to compilation issues.
Defects4J: 357 fault instances from five Java programs (JFreeChart, Closure, Commons‑Lang, Commons‑Math, Joda‑Time). 231 faults were used; 126 were excluded because test suite execution exceeded one hour per suite.
Each program comes with mature developer test suites. These were augmented with automatically generated tests:
For C: KLEE (symbolic execution, 22,208 tests) and manually written tests (166).
For Java: Randoop (random generation) and EvoSuite (covering branch coverage, weak mutation, and strong mutation), each run five times with a 300‑second budget per class, producing up to 2,000 tests per run (total 1,375,341 test cases).
Mutation Testing Tools
Mutants were generated using the same tools as Just et al. [24] and Chekam et al. [8], supporting common operators: AOR, LOR, COR, ROR, ORU, STD, and LVR. The analysis was also replicated with PIT (a separate Java mutation tool) and produced statistically indistinguishable results.
Sampling Methodology
For each fault, a large test pool was formed by merging developer and generated tests. Then:
10,000 test suites of random sizes were sampled (size range 0–20% of the pool).
Test suites were categorised as “Failing” (at least one test detects the fault) or “Passing” (all tests pass).
Mutation scores and test‑suite sizes were recorded.
Regression Analysis
Logistic regression models were fitted with fault detection (binary) as dependent variable and the following independent variables:
Mutation Score (MS)
Test‑suite size (Size)
Size × MS (interaction)
All three models were statistically significant (p < 0.05). The combined model (Size × MS) achieved significantly higher pseudo‑R² than either MS or Size alone (Wilcoxon signed‑rank test, p < 0.001), but overall predictive power remained moderate—fault detection cannot be predicted accurately from these variables alone.
Correlation Analysis
Pearson and Kendall correlation coefficients were computed between mutation score and fault detection under two conditions:
Uncontrolled size: Correlations were moderate to strong (0.35–0.75), consistent with Just et al.
Controlled size: Test suites were grouped by size (increments of 2.5% from 0% to 50% of the pool); within each group, correlations dropped to 0.05–0.20.
This demonstrates that the strong overall correlation is largely a product of test‑suite size. Results were consistent across all nine programs and both programming languages.
Fault Detection Probabilities
For each size group, test suites were sorted by mutation score. The top 10% (and top 25%) were compared against the entire set using a Chi‑squared test (α = 0.05). Statistically significant improvements in fault detection were observed for:
138 of 231 Defects4J faults (~60%)
59 of 68 CoREBench faults (~87%)
The remaining faults (40% for Defects4J, 13% for CoREBench) showed no evidence of improvement. These results indicate that, despite weak correlations, reaching the highest mutation scores does yield practical gains.
Figure 4 (described in text) plots the improvement for the top 10% versus all suites: most points lie above the diagonal, many with 95% confidence intervals that do not cross zero.
Relation Between Mutants and Real Faults
Only a tiny fraction (<1%) of the involved mutants represent the actual fault behaviour. The vast majority are “irrelevant”—they inflate mutation scores without contributing to real‑fault detection. This explains why mutation scores are poor proxies for test effectiveness even though they help guide test suite improvement.
Threats to Validity
Equivalent mutants remain undecidable; composed test pools with high‑coverage generation tools mitigate but do not eliminate this threat.
126 Defects4J faults were excluded due to computational infeasibility; sensitivity analysis using only developer test suites showed similar results.
The analysis used the fixed program version (following common practice); mutant behaviour may differ slightly on faulty versions, but previous work indicates minimal coupling variation.
Both Kendall and Pearson coefficients were reported; the use of biserial correlation (as in Just et al.) gave differences smaller than 0.001.
Conclusions
Test‑suite size and mutation score both influence fault detection, but the strong correlation between mutation score and fault detection is an artefact of their joint dependence on size.
When size is controlled, correlations are weak—mutants are not good substitutes for real faults in evaluating test effectiveness.
However, mutation testing remains valuable for improving test suites: achieving high mutation scores significantly raises the probability of detecting real faults.
Future work should focus on generating mutant sets that better represent real fault behaviour, reducing the noise from irrelevant mutants.