AssertFlip

Summary

AssertFlip is a technique for automatically generating Bug Reproducible Tests (BRTs) using large language models (LLMs). It targets the common situation in open-source and industrial settings where a reported bug has no executable test that reproduces it at the time of reporting, which makes diagnosis and repair slower and more difficult. Instead of asking an LLM to directly write a failing test, AssertFlip first generates passing tests against the buggy behavior and then inverts their assertions so they fail whenever the bug is present.

Key Points

• Bug reproduction is a critical step in debugging and repair, yet most reported bugs lack executable reproduction tests at report time.
• Existing methods try to generate failing tests directly; AssertFlip takes the opposite route: generate passing tests, then invert them.
• The core hypothesis is that LLMs are significantly better at writing passing tests than at deliberately writing tests that crash or fail.
• AssertFlip outperforms all known techniques on the SWT-Bench leaderboard, achieving a 43.6% fail-to-pass success rate on the SWT-Bench-Verified subset.

Concepts

• Bug Reproducible Tests (BRTs): executable tests that reproduce a reported bug by failing on the buggy code.
• Assertion inversion: the mechanism of converting a passing test's assertions into their logical opposites so the test fails when the buggy behavior is present.
• Fail-to-pass success rate: the evaluation metric measuring whether a generated test fails on the buggy version and passes once the bug is fixed.
• SWT-Bench: a benchmark curated specifically for evaluating bug reproducible test generation.

Details

The technique follows an inversion-based generation pipeline: