dorsal/arxiv
View SchemaAIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving
| Authors | Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, Junjie Lai |
|---|---|
| Categories | |
| ArXiv ID | 2601.06288vv1 |
| URL | https://arxiv.org/abs/2601.06288 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.
{
"annotation_id": "4fb7ec53-7f8b-4915-83b9-c078c862f66f",
"date_created": "2026-02-17T05:53:08.552000Z",
"date_modified": "2026-02-17T05:53:08.552000Z",
"file_hash": "05645ed3d8e6559cc2ca6cf2fcc88b9db81fd37e77006181c2ba45a9644ce822",
"private": false,
"record": {
"abstract": "Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.",
"arxiv_id": "2601.06288",
"authors": [
"Tianhao Xu",
"Yiming Liu",
"Xianglong Lu",
"Yijia Zhao",
"Xuting Zhou",
"Aichen Feng",
"Yiyi Chen",
"Yi Shen",
"Qin Zhou",
"Xumeng Chen",
"Ilya Sherstyuk",
"Haorui Li",
"Rishi Thakkar",
"Ben Hamm",
"Yuanzhe Li",
"Xue Huang",
"Wenpeng Wu",
"Anish Shanbhag",
"Harry Kim",
"Chuan Chen",
"Junjie Lai"
],
"categories": [
"cs.LG",
"cs.AI",
"cs.DC"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving",
"url": "https://arxiv.org/abs/2601.06288",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "24249f9d-c9cc-4eca-b1fb-2e43875e042e",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}