dorsal/arxiv
View SchemaRewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
| Authors | Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi |
|---|---|
| Categories | |
| ArXiv ID | 2601.08763vv1 |
| URL | https://arxiv.org/abs/2601.08763 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.
{
"annotation_id": "26fbd3c9-5e97-474d-ba71-bb06fde4837f",
"date_created": "2026-02-17T05:53:16.080000Z",
"date_modified": "2026-02-17T05:53:16.080000Z",
"file_hash": "41039475569f7c760f79a2d6de3b18c6a4ee40f8dc687db8d79229909ceb8ef1",
"private": false,
"record": {
"abstract": "Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.",
"arxiv_id": "2601.08763",
"authors": [
"Zhiyuan Hu",
"Yucheng Wang",
"Yufei He",
"Jiaying Wu",
"Yilun Zhao",
"See-Kiong Ng",
"Cynthia Breazeal",
"Anh Tuan Luu",
"Hae Won Park",
"Bryan Hooi"
],
"categories": [
"cs.LG",
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs",
"url": "https://arxiv.org/abs/2601.08763",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "006517a6-f82a-4d49-afb8-55694a154336",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}