dorsal/arxiv
View SchemaCan Large Language Models Understand, Reason About, and Generate Code-Switched Text?
| Authors | Genta Indra Winata, David Anugraha, Patrick Amadeus Irawan, Anirban Das, Haneul Yoo, Paresh Dashore, Shreyas Kulkarni, Ruochen Zhang, Haruki Sakajo, Frederikus Hudi, Anaelia Ovalle, Syrielle Montariol, Felix Gaschi, Michael Anugraha, Rutuj Ravindra Puranik, Zawad Hayat Ahmed, Adril Putra Merin, Emmanuele Chersoni |
|---|---|
| Categories | |
| ArXiv ID | 2601.07153vv1 |
| URL | https://arxiv.org/abs/2601.07153 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source.
{
"annotation_id": "3cdb394c-f743-46ae-80fc-5c504b21d51b",
"date_created": "2026-02-17T05:53:12.606000Z",
"date_modified": "2026-02-17T05:53:12.606000Z",
"file_hash": "71ce87d970a9f14e0ce0d84d32292a36e299b9314d7b8c64051e3949398105ce",
"private": false,
"record": {
"abstract": "Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source.",
"arxiv_id": "2601.07153",
"authors": [
"Genta Indra Winata",
"David Anugraha",
"Patrick Amadeus Irawan",
"Anirban Das",
"Haneul Yoo",
"Paresh Dashore",
"Shreyas Kulkarni",
"Ruochen Zhang",
"Haruki Sakajo",
"Frederikus Hudi",
"Anaelia Ovalle",
"Syrielle Montariol",
"Felix Gaschi",
"Michael Anugraha",
"Rutuj Ravindra Puranik",
"Zawad Hayat Ahmed",
"Adril Putra Merin",
"Emmanuele Chersoni"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?",
"url": "https://arxiv.org/abs/2601.07153",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "c282f553-75bc-472e-b9dc-b7529ffe81f1",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}