dorsal/arxiv
View SchemaTurkBench: A Benchmark for Evaluating Turkish Large Language Models
| Authors | Çağrı Toraman, Ahmet Kaan Sever, Ayse Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Osman Gürlek, Sarp Kantar, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, Onur Gungor, Susan Üsküdarlı, Abdullah Topraksoy, Esra Darıcı |
|---|---|
| Categories | |
| ArXiv ID | 2601.07020vv1 |
| URL | https://arxiv.org/abs/2601.07020 |
| License | http://creativecommons.org/licenses/by-nc-sa/4.0/ |
Abstract
With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench
{
"annotation_id": "33942a59-fbc1-4fbb-bfac-7692bbbd760a",
"date_created": "2026-02-17T05:53:08.668000Z",
"date_modified": "2026-02-17T05:53:08.668000Z",
"file_hash": "fe6f1ae9302de78b9d75ae9f85e921dcd9f5520211bb5ed5d18a00db9ec039ce",
"private": false,
"record": {
"abstract": "With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench",
"arxiv_id": "2601.07020",
"authors": [
"\u00c7a\u011fr\u0131 Toraman",
"Ahmet Kaan Sever",
"Ayse Aysu Cengiz",
"Elif Ecem Arslan",
"G\u00f6rkem Sevin\u00e7",
"Mete Mert Birdal",
"Yusuf Faruk G\u00fcldemir",
"Ali Bu\u011fra Kanburo\u011flu",
"Sezen Feleko\u011flu",
"Osman G\u00fcrlek",
"Sarp Kantar",
"Birsen \u015eahin K\u00fct\u00fck",
"B\u00fc\u015fra Tufan",
"Elif Gen\u00e7",
"Serkan Co\u015fkun",
"Gupse Ekin Demir",
"Muhammed Emin Aray\u0131c\u0131",
"Olgun Dursun",
"Onur Gungor",
"Susan \u00dcsk\u00fcdarl\u0131",
"Abdullah Topraksoy",
"Esra Dar\u0131c\u0131"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
"title": "TurkBench: A Benchmark for Evaluating Turkish Large Language Models",
"url": "https://arxiv.org/abs/2601.07020",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "d0592b14-a4a4-45f8-afdb-f6b0209523dc",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}