dorsal/arxiv
View SchemaConsolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration
| Authors | Yang Zhao, Yangou Ouyang, Xiao Ding, Hepeng Wang, Bibo Cai, Kai Xiong, Jinglong Gao, Zhouhao Sun, Li Du, Bing Qin, Ting Liu |
|---|---|
| Categories | |
| ArXiv ID | 2601.07224vv1 |
| URL | https://arxiv.org/abs/2601.07224 |
| License | http://creativecommons.org/licenses/by-nc-nd/4.0/ |
Abstract
While Hybrid Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become the standard paradigm for training LLM agents, effective mechanisms for data allocation between these stages remain largely underexplored. Current data arbitration strategies often rely on surface-level heuristics that fail to diagnose intrinsic learning needs. Since SFT targets pattern consolidation through imitation while RL drives structural adaptation via exploration, misaligning data with these functional roles causes severe optimization interference. We propose PRISM, a dynamics-aware framework grounded in Schema Theory that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge. By analyzing the spatial geometric structure of gradients, PRISM identifies data triggering high spatial concentration as high-conflict signals that require RL for structural restructuring. In contrast, data yielding diffuse updates is routed to SFT for efficient consolidation. Extensive experiments on WebShop and ALFWorld demonstrate that PRISM achieves a Pareto improvement, outperforming state-of-the-art hybrid methods while reducing computational costs by up to 3.22$\times$. Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
{
"annotation_id": "517525f4-0f09-424b-a205-4433f7977fbd",
"date_created": "2026-02-17T05:53:11.760000Z",
"date_modified": "2026-02-17T05:53:11.760000Z",
"file_hash": "568fa4896b04dafb710caa78a2a4138008396085c2a539ba7c39c00b9d0ff672",
"private": false,
"record": {
"abstract": "While Hybrid Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become the standard paradigm for training LLM agents, effective mechanisms for data allocation between these stages remain largely underexplored. Current data arbitration strategies often rely on surface-level heuristics that fail to diagnose intrinsic learning needs. Since SFT targets pattern consolidation through imitation while RL drives structural adaptation via exploration, misaligning data with these functional roles causes severe optimization interference. We propose PRISM, a dynamics-aware framework grounded in Schema Theory that arbitrates data based on its degree of cognitive conflict with the model\u0027s existing knowledge. By analyzing the spatial geometric structure of gradients, PRISM identifies data triggering high spatial concentration as high-conflict signals that require RL for structural restructuring. In contrast, data yielding diffuse updates is routed to SFT for efficient consolidation. Extensive experiments on WebShop and ALFWorld demonstrate that PRISM achieves a Pareto improvement, outperforming state-of-the-art hybrid methods while reducing computational costs by up to 3.22$\\times$. Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.",
"arxiv_id": "2601.07224",
"authors": [
"Yang Zhao",
"Yangou Ouyang",
"Xiao Ding",
"Hepeng Wang",
"Bibo Cai",
"Kai Xiong",
"Jinglong Gao",
"Zhouhao Sun",
"Li Du",
"Bing Qin",
"Ting Liu"
],
"categories": [
"cs.AI",
"cs.LG"
],
"license": "http://creativecommons.org/licenses/by-nc-nd/4.0/",
"title": "Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration",
"url": "https://arxiv.org/abs/2601.07224",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "a5e43b63-4b61-4726-8d5b-bc06c491289a",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}