Summary
This article covers a large-scale replication study (accepted at ISSTA 2026) that re-examines whether code coverage and mutation score — standard proxy metrics for test quality — are valid indicators of real-bug detection when test suites are generated by large language models (LLMs). The study replicates prior work by Inozemtseva et al. and Papadakis et al., which showed that for human-written tests, correlations among coverage, mutation, and real-bug detection largely vanish once test suite size is controlled. The replication finds that these conclusions do not carry over cleanly to LLM-generated tests: the usefulness of coverage and mutation is highly context-dependent, and test suite size is not a dominant confounder in this setting.
Key Points
- Replicates two influential prior studies in the context of LLM-generated test suites, using a diverse set of LLMs and a wide range of generated suites.
- Findings diverge substantially from the human-written-test results of Inozemtseva et al. and Papadakis et al.
- Coverage and mutation metrics are meaningful signals in regression-style settings — where the code given to the LLM can reasonably be assumed bug-free — especially when comparing across models.
- Coverage and mutation are not reliable indicators when the code-under-test may already be buggy and the goal is to expose that bug.
- Little evidence was found that test suite size is a dominant confounding factor for correlations among coverage, mutation, and real-bug detection in LLM-generated tests.
- Provides actionable guidance for designing and interpreting evaluations of LLM-based test generation.
Concepts
- Proxy metrics: Indirect measures of test quality — here, code coverage and mutation score — used in place of directly measuring real-bug detection effectiveness.
- Code coverage: The proportion of code exercised by a test suite; commonly used as a quick quality signal.
- Mutation score: The percentage of mutated (fault-injected) program variants detected by a test suite; a stronger proxy than coverage.