dorsal/arxiv
View SchemaCHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space
| Authors | Bingyi Liu, Jinbo He, Haiyong Shi, Enshu Wang, Weizhen Han, Jingxiang Hao, Peixi Wang, Zhuangzhuang Zhang |
|---|---|
| Categories | |
| ArXiv ID | 2601.05675vv1 |
| URL | https://arxiv.org/abs/2601.05675 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Hybrid action space, which combines discrete choices and continuous parameters, is prevalent in domains such as robot control and game AI. However, efficiently modeling and optimizing hybrid discrete-continuous action space remains a fundamental challenge, mainly due to limited policy expressiveness and poor scalability in high-dimensional settings. To address this challenge, we view the hybrid action space problem as a fully cooperative game and propose a \textbf{Cooperative Hybrid Diffusion Policies (CHDP)} framework to solve it. CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively. The continuous policy is conditioned on the discrete action's representation, explicitly modeling the dependency between them. This cooperative design allows the diffusion policies to leverage their expressiveness to capture complex distributions in their respective action spaces. To mitigate the update conflicts arising from simultaneous policy updates in this cooperative setting, we employ a sequential update scheme that fosters co-adaptation. Moreover, to improve scalability when learning in high-dimensional discrete action space, we construct a codebook that embeds the action space into a low-dimensional latent space. This mapping enables the discrete policy to learn in a compact, structured space. Finally, we design a Q-function-based guidance mechanism to align the codebook's embeddings with the discrete policy's representation during training. On challenging hybrid action benchmarks, CHDP outperforms the state-of-the-art method by up to $19.3\%$ in success rate.
{
"annotation_id": "a7c2e2c1-b1c4-42aa-9868-751fae8e4a0b",
"date_created": "2026-02-17T05:53:05.056000Z",
"date_modified": "2026-02-17T05:53:05.056000Z",
"file_hash": "0313caba4ee017167a6a845448c0553a34e8ce121943a1f20f1eb4e0c2519fc4",
"private": false,
"record": {
"abstract": "Hybrid action space, which combines discrete choices and continuous parameters, is prevalent in domains such as robot control and game AI. However, efficiently modeling and optimizing hybrid discrete-continuous action space remains a fundamental challenge, mainly due to limited policy expressiveness and poor scalability in high-dimensional settings. To address this challenge, we view the hybrid action space problem as a fully cooperative game and propose a \\textbf{Cooperative Hybrid Diffusion Policies (CHDP)} framework to solve it. CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively. The continuous policy is conditioned on the discrete action\u0027s representation, explicitly modeling the dependency between them. This cooperative design allows the diffusion policies to leverage their expressiveness to capture complex distributions in their respective action spaces. To mitigate the update conflicts arising from simultaneous policy updates in this cooperative setting, we employ a sequential update scheme that fosters co-adaptation. Moreover, to improve scalability when learning in high-dimensional discrete action space, we construct a codebook that embeds the action space into a low-dimensional latent space. This mapping enables the discrete policy to learn in a compact, structured space. Finally, we design a Q-function-based guidance mechanism to align the codebook\u0027s embeddings with the discrete policy\u0027s representation during training. On challenging hybrid action benchmarks, CHDP outperforms the state-of-the-art method by up to $19.3\\%$ in success rate.",
"arxiv_id": "2601.05675",
"authors": [
"Bingyi Liu",
"Jinbo He",
"Haiyong Shi",
"Enshu Wang",
"Weizhen Han",
"Jingxiang Hao",
"Peixi Wang",
"Zhuangzhuang Zhang"
],
"categories": [
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space",
"url": "https://arxiv.org/abs/2601.05675",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "55632dd6-db69-42e6-911a-10905674b25e",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}