SWE-Mutation is a benchmark for evaluating the quality of LLM-generated test suites in software engineering. It addresses a core bottleneck in scaling LLM software-engineering evaluation: not the scarcity of high-quality solutions, but the scarcity of high-quality test suites. The benchmark probes test suites with systematically mutated (buggy) program variants that attempt to "fool" the suite and pass validation. It also introduces an agentic, language-agnostic framework for automatically generating complex, realistic mutants. Experiments on seven LLMs show that even a strong model such as DeepSeek-V3.1 achieves only 10.20% verification and 36.15% detection rates, exposing persistent weaknesses in the ability of current LLMs to produce reliable, discriminative test suites.
Key Points
High-quality test suites are the key bottleneck for LLM software-engineering evaluation; they are needed both for synthesizing program-repair trajectories and for providing precise reward signals in reinforcement learning.
Manually annotated test suites are expensive and scarce, while LLM-generated tests tend to be superficial and lack discriminative power.
SWE-Mutation characterizes test suites by introducing mutated solutions designed to "fool" them and pass validation — a mutation-testing-inspired approach.
The benchmark contains 2,636 mutated variants derived from 800 original instances, including a multilingual subset spanning nine programming languages.
A novel agentic, language-agnostic mutation framework generates complex mutants that are harder for test suites to detect than conventionally generated ones.
Agentic mutants reduced average detection rates from 71.04% (conventional methods) to 39.81%, indicating greater realism and challenge.
Across seven evaluated LLMs, current models still cannot generate test suites that are both reliable and sufficiently discriminative.
Concepts
Test-suite quality / discriminative power: The ability of a test suite to accept correct solutions while rejecting incorrect (mutated) ones.
Mutation testing: A technique that evaluates tests by seeding faults (mutants) into a program and measuring whether the tests detect them.
Verification rate: The rate at which a candidate test suite validates the correct (reference) solution — a low rate means the generated tests are unreliable even on correct code.
Detection rate: The rate at which a candidate test suite detects mutated (buggy) variants — a low rate means the tests lack discriminative power.
Agentic mutation framework: An LLM-agent-driven, language-agnostic pipeline for automatically generating complex, realistic mutants.
Multilingual subset: A portion of the benchmark covering nine programming languages, used to test language-agnostic generalization.
Details
Motivation: the test-suite bottleneck
Evaluating software-engineering capability has become a standard part of LLM development. However, scaling this evaluation is blocked less by the supply of high-quality problem solutions than by the lack of high-quality test suites. Test suites serve two essential roles: they enable synthesis of program-repair trajectories, and they provide precise feedback signals for reinforcement learning. Because high-quality test suites are costly and difficult to annotate, they have long been scarce; test suites generated automatically by LLMs tend to be shallow and lack the discriminative power needed to separate correct from incorrect programs.
Benchmark design
SWE-Mutation applies mutation-testing logic to the problem of evaluating test suites. Instead of judging tests only against correct solutions, the benchmark constructs systematic mutations of solutions that attempt to "fool" a test suite and pass validation. A test suite's quality is then characterized by how well it resists these mutants:
Verification captures whether the test suite accepts the correct reference solution.
Detection captures whether the test suite rejects mutated, buggy variants.
A reliable and discriminative test suite must score high on both.
Agentic mutant generation
To make the benchmark realistic, the authors propose an agentic, language-agnostic framework that automatically generates complex mutants. Unlike conventional mutation operators, which tend to produce mutants that are easy for tests to catch, the agentic approach produces mutations that more closely resemble naturally occurring bugs. When tested against LLM-generated test suites, these agentic mutants cut average detection rates from 71.04% to 39.81%, demonstrating that they are substantially harder to detect and therefore more realistic evaluation material.
Scale and multilingual coverage
The benchmark comprises 2,636 mutated variants derived from 800 original instances. A multilingual subset spans nine programming languages, enabling assessment of whether test-suite generation quality generalizes beyond a single language.
Experimental findings
Across seven LLMs, even the strongest model evaluated (DeepSeek-V3.1) achieved only 10.20% verification and 36.15% detection rates. The low verification rate indicates that generated test suites frequently fail on correct solutions, while the low detection rate shows they miss most mutated variants. Together, these results highlight that current LLMs are far from generating test suites that are both reliable and discriminative, and point to test-suite generation as a critical open problem for scaling LLM software-engineering evaluation.