Summary
This study characterizes the oracle signals present in test code written by coding agents in pull requests, using the AIDev-pop dataset. The corpus covers 33,596 PRs across 2,807 GitHub repositories (each with at least 100 stars) and five coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. From 711,923 total file patches, the authors extracted 103,976 test code patches, which were aggregated into 86,156 cumulative test-file patches. The central finding is that 80.2% of agent-authored test-file patches contain weak or no explicit oracle signals, meaning most AI-generated tests execute code paths without actually verifying output correctness. After controlling for confounders, strong oracles are nonetheless associated with significantly higher merge likelihood, and test-file counts substantially overestimate the verification strength of agent-authored contributions.
Key Points
- 80.2% of agent-authored test-file patches have weak or no explicit oracle signals; value assertions (S1) appear in 11.3% of patches and multi-signal strong oracles (S3) in 5.7%.
- Agent behavior differs significantly (χ² = 2497.3, p < 0.001): Claude Code and Devin produce stronger oracle profiles than Copilot, Cursor, and OpenAI Codex.
- Newly created test files show higher strong-oracle rates than modified files (χ² = 810.2, p < 0.001), ranging from 18% for OpenAI Codex to 67% for Claude Code.
- Raw merge rates are lower for strong-oracle PRs: S3 PRs merge at 59.7% versus 72.6% for weak-oracle PRs.
- S3 PRs contain 4.2× more code additions (Mann-Whitney, p < 0.001, r = 0.38), attract 2.4× more review effort (Mann-Whitney, p < 0.001, r = 0.36), and reside in repositories with 3.8× more stars.
- After regression adjustment for agent, PR size, repository popularity, task type, and language, strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001).
- Review-effort gaps persist within PR size buckets: 1.7× for small PRs (1–5 files), up to 2.5× for large PRs (>20 files).
Concepts
- Test oracle: The assertion or check that determines whether a test passes or fails. Without an explicit oracle, a test may execute code without verifying behavior.