dorsal/arxiv
View SchemaRULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation
| Authors | Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong |
|---|---|
| Categories | |
| ArXiv ID | 2601.08654vv1 |
| URL | https://arxiv.org/abs/2601.08654 |
| License | http://creativecommons.org/publicdomain/zero/1.0/ |
Abstract
The LLM-as-a-Judge paradigm promises scalable rubric-based evaluation, yet aligning frozen black-box models with human standards remains a challenge due to inherent generation stochasticity. We reframe judge alignment as a criteria transfer problem and isolate three recurrent failure modes: rubric instability caused by prompt sensitivity, unverifiable reasoning that lacks auditable evidence, and scale misalignment with human grading boundaries. To address these issues, we introduce RULERS (Rubric Unification, Locking, and Evidence-anchored Robust Scoring), a compiler-executor framework that transforms natural language rubrics into executable specifications. RULERS operates by compiling criteria into versioned immutable bundles, enforcing structured decoding with deterministic evidence verification, and applying lightweight Wasserstein-based post-hoc calibration, all without updating model parameters. Extensive experiments on essay and summarization benchmarks demonstrate that RULERS significantly outperforms representative baselines in human agreement, maintains strong stability against adversarial rubric perturbations, and enables smaller models to rival larger proprietary judges. Overall, our results suggest that reliable LLM judging requires executable rubrics, verifiable evidence, and calibrated scales rather than prompt phrasing alone. Code is available at https://github.com/LabRAI/Rulers.git.
{
"annotation_id": "13119894-5ebc-49f7-b933-f5c2006ee854",
"date_created": "2026-02-17T05:53:16.291000Z",
"date_modified": "2026-02-17T05:53:16.291000Z",
"file_hash": "e970e469524c769ef366d4731ea745a6425b5247b991ca6c8c4165631245c18c",
"private": false,
"record": {
"abstract": "The LLM-as-a-Judge paradigm promises scalable rubric-based evaluation, yet aligning frozen black-box models with human standards remains a challenge due to inherent generation stochasticity. We reframe judge alignment as a criteria transfer problem and isolate three recurrent failure modes: rubric instability caused by prompt sensitivity, unverifiable reasoning that lacks auditable evidence, and scale misalignment with human grading boundaries. To address these issues, we introduce RULERS (Rubric Unification, Locking, and Evidence-anchored Robust Scoring), a compiler-executor framework that transforms natural language rubrics into executable specifications. RULERS operates by compiling criteria into versioned immutable bundles, enforcing structured decoding with deterministic evidence verification, and applying lightweight Wasserstein-based post-hoc calibration, all without updating model parameters. Extensive experiments on essay and summarization benchmarks demonstrate that RULERS significantly outperforms representative baselines in human agreement, maintains strong stability against adversarial rubric perturbations, and enables smaller models to rival larger proprietary judges. Overall, our results suggest that reliable LLM judging requires executable rubrics, verifiable evidence, and calibrated scales rather than prompt phrasing alone. Code is available at https://github.com/LabRAI/Rulers.git.",
"arxiv_id": "2601.08654",
"authors": [
"Yihan Hong",
"Huaiyuan Yao",
"Bolin Shen",
"Wanpeng Xu",
"Hua Wei",
"Yushun Dong"
],
"categories": [
"cs.CL",
"cs.AI",
"cs.LG"
],
"license": "http://creativecommons.org/publicdomain/zero/1.0/",
"title": "RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation",
"url": "https://arxiv.org/abs/2601.08654",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "d16fae68-4266-4ebf-8bd2-a78a90e89fb9",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}