dorsal/arxiv
View SchemaMoCha:End-to-End Video Character Replacement without Structural Guidance
| Authors | Zhengbo Xu, Jie Ma, Ziheng Wang, Zhan Peng, Jun Liang, Jing Li |
|---|---|
| Categories | |
| ArXiv ID | 2601.08587vv2 |
| URL | https://arxiv.org/abs/2601.08587 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
{
"annotation_id": "e9427ca1-37d7-4351-a8ca-7c57267383fb",
"date_created": "2026-02-17T05:53:16.246000Z",
"date_modified": "2026-02-17T05:53:16.246000Z",
"file_hash": "e78c74dfae6e4a0a0f362f5849820129a4f55ac00cbf9048d9607fdf145a1a4f",
"private": false,
"record": {
"abstract": "Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha",
"arxiv_id": "2601.08587",
"authors": [
"Zhengbo Xu",
"Jie Ma",
"Ziheng Wang",
"Zhan Peng",
"Jun Liang",
"Jing Li"
],
"categories": [
"cs.CV"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "MoCha:End-to-End Video Character Replacement without Structural Guidance",
"url": "https://arxiv.org/abs/2601.08587",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "315d808a-b440-4be4-916e-31c92da57cf5",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}