dorsal/arxiv
View SchemaSpeech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception
| Authors | Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Ehsan Hosseini Asl, Chenhui Chu, Shinji Watanabe, Yu-Chiang Frank Wang, Boris Ginsburg |
|---|---|
| Categories | |
| ArXiv ID | 2601.09413vv1 |
| URL | https://arxiv.org/abs/2601.09413 |
| License | http://creativecommons.org/licenses/by-nc-sa/4.0/ |
Abstract
We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, Speech-Hands consistently outperforms strong baselines by 12.1% WER on seven benchmarks. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence.
{
"annotation_id": "4c8c4487-ace0-445f-8486-3e294c985890",
"date_created": "2026-02-17T05:53:20.329000Z",
"date_modified": "2026-02-17T05:53:20.329000Z",
"file_hash": "46e1a156e1485c3c841ad525baf76d4ec36f064a5561ad1c27766c4742122264",
"private": false,
"record": {
"abstract": "We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, Speech-Hands consistently outperforms strong baselines by 12.1% WER on seven benchmarks. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence.",
"arxiv_id": "2601.09413",
"authors": [
"Zhen Wan",
"Chao-Han Huck Yang",
"Jinchuan Tian",
"Hanrong Ye",
"Ankita Pasad",
"Szu-wei Fu",
"Arushi Goel",
"Ryo Hachiuma",
"Shizhe Diao",
"Kunal Dhawan",
"Sreyan Ghosh",
"Yusuke Hirota",
"Zhehuai Chen",
"Rafael Valle",
"Ehsan Hosseini Asl",
"Chenhui Chu",
"Shinji Watanabe",
"Yu-Chiang Frank Wang",
"Boris Ginsburg"
],
"categories": [
"cs.SD",
"cs.AI",
"cs.CL",
"cs.MA",
"eess.AS"
],
"license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
"title": "Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception",
"url": "https://arxiv.org/abs/2601.09413",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "e62954e1-a8ce-4bc1-b949-b83c183832d0",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}