dorsal/arxiv
View SchemaVideo Generation Models in Robotics -- Applications, Research Challenges, Future Directions
| Authors | Zhiting Mei, Tenny Yin, Ola Shorinwa, Apurva Badithela, Zhonghe Zheng, Joseph Bruno, Madison Bland, Lihan Zha, Asher Hancock, Jaime Fernández Fisac, Philip Dames, Anirudha Majumdar |
|---|---|
| Categories | |
| ArXiv ID | 2601.07823vv1 |
| URL | https://arxiv.org/abs/2601.07823 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Video generation models have emerged as high-fidelity models of the physical world, capable of synthesizing high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models enable photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. In this survey, we provide a review of video models and their applications as embodied world models in robotics, encompassing cost-effective data generation and action prediction in imitation learning, dynamics and rewards modeling in reinforcement learning, visual planning, and policy evaluation. Further, we highlight important challenges hindering the trustworthy integration of video models in robotics, which include poor instruction following, hallucinations such as violations of physics, and unsafe content generation, in addition to fundamental limitations such as significant data curation, training, and inference costs. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings.
{
"annotation_id": "faae38fd-3a93-4b1a-b81b-e824017b0457",
"date_created": "2026-02-17T05:53:11.938000Z",
"date_modified": "2026-02-17T05:53:11.938000Z",
"file_hash": "ed08977b29f65a122b397f14462deeb0299914b9ffa38d2ad3e6e7b265a228f2",
"private": false,
"record": {
"abstract": "Video generation models have emerged as high-fidelity models of the physical world, capable of synthesizing high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models enable photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. In this survey, we provide a review of video models and their applications as embodied world models in robotics, encompassing cost-effective data generation and action prediction in imitation learning, dynamics and rewards modeling in reinforcement learning, visual planning, and policy evaluation. Further, we highlight important challenges hindering the trustworthy integration of video models in robotics, which include poor instruction following, hallucinations such as violations of physics, and unsafe content generation, in addition to fundamental limitations such as significant data curation, training, and inference costs. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings.",
"arxiv_id": "2601.07823",
"authors": [
"Zhiting Mei",
"Tenny Yin",
"Ola Shorinwa",
"Apurva Badithela",
"Zhonghe Zheng",
"Joseph Bruno",
"Madison Bland",
"Lihan Zha",
"Asher Hancock",
"Jaime Fern\u00e1ndez Fisac",
"Philip Dames",
"Anirudha Majumdar"
],
"categories": [
"eess.SY",
"cs.RO",
"cs.SY"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions",
"url": "https://arxiv.org/abs/2601.07823",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "964e2119-cee4-4f5b-b88e-abbbdfda3a5c",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}