dorsal/arxiv
View SchemaSLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
| Authors | Ziyang Ma, Guanrou Yang, Wenxi Chen, Zhifu Gao, Yexing Du, Xiquan Li, Zhisheng Zheng, Haina Zhu, Jianheng Zhuo, Zheshu Song, Ruiyang Xu, Tiranrui Wang, Yifan Yang, Yanqiao Zhu, Zhikang Niu, Liumeng Xue, Yinghao Ma, Ruibin Yuan, Shiliang Zhang, Kai Yu, Eng Siong Chng, Xie Chen |
|---|---|
| Categories | |
| ArXiv ID | 2601.09385vv1 |
| URL | https://arxiv.org/abs/2601.09385 |
| DOI | 10.1109/JSTSP.2026.3653157 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
{
"annotation_id": "3e47f865-8979-405b-9646-d4c2bce7b1f9",
"date_created": "2026-02-17T05:53:19.911000Z",
"date_modified": "2026-02-17T05:53:19.911000Z",
"file_hash": "789c644a55c334b4e86583f456805b3cdd6912acdc8ac512fef5a4785a9facb2",
"private": false,
"record": {
"abstract": "The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.",
"arxiv_id": "2601.09385",
"authors": [
"Ziyang Ma",
"Guanrou Yang",
"Wenxi Chen",
"Zhifu Gao",
"Yexing Du",
"Xiquan Li",
"Zhisheng Zheng",
"Haina Zhu",
"Jianheng Zhuo",
"Zheshu Song",
"Ruiyang Xu",
"Tiranrui Wang",
"Yifan Yang",
"Yanqiao Zhu",
"Zhikang Niu",
"Liumeng Xue",
"Yinghao Ma",
"Ruibin Yuan",
"Shiliang Zhang",
"Kai Yu",
"Eng Siong Chng",
"Xie Chen"
],
"categories": [
"cs.SD",
"cs.CL",
"cs.MM"
],
"doi": "10.1109/JSTSP.2026.3653157",
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing",
"url": "https://arxiv.org/abs/2601.09385",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "d92c766d-0694-4f41-8483-a4269963094b",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}