dorsal/arxiv
View SchemaCross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models
| Authors | Haorui Yu, Ramon Ruiz-Dolz, Xuehang Wen, Fengrui Zhang, Qiufeng Yi |
|---|---|
| Categories | |
| ArXiv ID | 2601.07984vv1 |
| URL | https://arxiv.org/abs/2601.07984 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated. We present a tri-tier evaluation framework for cross-cultural art-critique assessment: Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions; and Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set. The framework outputs a calibrated cultural-understanding score for model selection and cultural-gap diagnosis, together with dimension-level diagnostics and risk indicators. We evaluate 15 VLMs on 294 expert anchors spanning six cultural traditions. Key findings are that (i) automated metrics are unreliable proxies for cultural depth, (ii) Western samples score higher than non-Western samples under our sampling and rubric, and (iii) cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration. Dataset and code are available in the supplementary materials.
{
"annotation_id": "0a6e1974-e4f8-4882-9664-d12f0000f81d",
"date_created": "2026-02-17T05:53:16.296000Z",
"date_modified": "2026-02-17T05:53:16.296000Z",
"file_hash": "903732fb9e753921ff25536b8c62ee4edffddb591d7fcff93f3e01f9acfce37b",
"private": false,
"record": {
"abstract": "Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated. We present a tri-tier evaluation framework for cross-cultural art-critique assessment: Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions; and Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set. The framework outputs a calibrated cultural-understanding score for model selection and cultural-gap diagnosis, together with dimension-level diagnostics and risk indicators. We evaluate 15 VLMs on 294 expert anchors spanning six cultural traditions. Key findings are that (i) automated metrics are unreliable proxies for cultural depth, (ii) Western samples score higher than non-Western samples under our sampling and rubric, and (iii) cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration. Dataset and code are available in the supplementary materials.",
"arxiv_id": "2601.07984",
"authors": [
"Haorui Yu",
"Ramon Ruiz-Dolz",
"Xuehang Wen",
"Fengrui Zhang",
"Qiufeng Yi"
],
"categories": [
"cs.CL"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models",
"url": "https://arxiv.org/abs/2601.07984",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "58667d00-be8e-42b9-bb48-45cac0a2c913",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}