Summary
An oracle signal is the verdict-producing element of a test — the assertion (or its absence) that lets the test distinguish correct from incorrect behavior. A test file can exercise code extensively yet contain little or no oracle signal, so test presence substantially overestimates verification strength. A 2026 line of research on agent-authored tests found that roughly 80% of AI-generated test-file patches contain weak or no explicit oracle signals, and that suites may pass execution without carrying semantic value ("stealth technical debt").
The standard instrument for measuring whether a suite can detect faults is mutation testing — seeding defective variants (mutants) of the code and seeing if the suite fails them. Its usefulness is context-dependent, and surviving conventional mutants is only a floor, not a ceiling. Complementary tooling, such as greenproof, detects "unearned" green results when the same session that edited the code also authored the test that blesses it.
Key Points
- Test-file presence overestimates verification: 80.2% of 86,156 agent-authored test-file patches (33,596 PRs, 5 agents) had weak or no explicit oracle signals (All Smoke, No Alarm).
- Agent-generated tests beat humans on edge-case variety (0.62 vs 0.32) but are slightly weaker on assertion strength (85.x% vs 88.1%) — "stealth technical debt": suites that pass execution without semantic value (Beyond Test Presence).
- Coverage and mutation scores are meaningful in regression-style settings — where the code is assumed bug-free and the question is whether the suite would catch a future change that breaks it — but not when the goal is exposing an existing bug (ISSTA 2026).
- Conventional operator mutants are the easy set: agentic mutants cut detection from 71.0% to 39.8%. A good survival rate is therefore a floor, not a ceiling; failing it is damning, passing it proves little (SWE-Mutation).
- Eval blind spots are gaps in , not model capability — a stronger model never closes them; only a change to methodology does.