Summary
ReVeal is a multi-turn reinforcement learning framework that improves code generation by explicitly training large language models to verify their own outputs. It extends reinforcement learning with verifiable rewards (RLVR) by pairing generation with self-verification and tool-based evaluation, enabling models to iteratively refine code at inference time. The framework is designed to make self-verification a reliable driver of test-time scaling.
Key Points
- Addresses a key limitation of RLVR: relying only on outcome rewards without optimizing verification.
- Structures long-horizon reasoning as iterative generation-verification turns.
- Uses TAPO for turn-level credit assignment across multi-turn trajectories.
- Promotes co-evolution of code generation and test generation.
- Models trained on only three turns continue to improve for 20+ turns at inference on LiveCodeBench.
- Significantly improves Pass@k, indicating stronger exploration and broader reasoning.
Concepts
- RLVR: Reinforcement learning with verifiable rewards, used to train reasoning in LLMs by rewarding correct, verifiable outcomes.
- Self-verification: The model’s ability to construct tests and use tool feedback to check and refine its own generated code.
- Verification-generation asymmetry: The imbalance between a model’s capacity to generate outputs and its capacity to verify them. ReVeal strengthens verification to reduce this gap.
- TAPO: Turn-level advantage policy optimization, the credit assignment method used to assign rewards to individual turns within a long-horizon trajectory.
- Test-time scaling: Using additional computation during inference to improve output quality; in ReVeal, this takes the form of repeated generation-verification loops.
Details
Existing RLVR methods typically optimize only the final outcome reward. They do not explicitly train the model to verify its outputs, which leads to unreliable self-verification and limits test-time scaling. ReVeal is designed to overcome this by making self-verification a first-class training objective.