dorsal/arxiv
View SchemaJudgeRLVR: Judge First, Generate Second for Efficient Reasoning
| Authors | Jiangshan Duo, Hanyu Li, Hailin Zhang, Yudong Wang, Sujian Li, Liang Zhao |
|---|---|
| Categories | |
| ArXiv ID | 2601.08468vv1 |
| URL | https://arxiv.org/abs/2601.08468 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality--efficiency trade-off for Qwen3-30B-A3B: on in-domain math, it delivers about +3.7 points average accuracy gain with -42\% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization.
{
"annotation_id": "c0559b6e-9736-4842-85c1-2f3fea81f34e",
"date_created": "2026-02-17T05:53:16.007000Z",
"date_modified": "2026-02-17T05:53:16.007000Z",
"file_hash": "11495775d55bc9d8b194d6326ed277f55e3cce951b0c173d1c1ee154dbeb08d5",
"private": false,
"record": {
"abstract": "Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality--efficiency trade-off for Qwen3-30B-A3B: on in-domain math, it delivers about +3.7 points average accuracy gain with -42\\% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization.",
"arxiv_id": "2601.08468",
"authors": [
"Jiangshan Duo",
"Hanyu Li",
"Hailin Zhang",
"Yudong Wang",
"Sujian Li",
"Liang Zhao"
],
"categories": [
"cs.CL",
"cs.AI",
"cs.LG"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "JudgeRLVR: Judge First, Generate Second for Efficient Reasoning",
"url": "https://arxiv.org/abs/2601.08468",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "2e70faa2-50bb-4b91-8d73-fb49f7095671",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}