dorsal/arxiv
View SchemaWhat Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models
| Authors | Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Ahn Eungyeol, Jungwhan Kim, Seunghyeok Hong, Youngsook Song |
|---|---|
| Categories | |
| ArXiv ID | 2601.06165vv1 |
| URL | https://arxiv.org/abs/2601.06165 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.
{
"annotation_id": "d290c377-00e5-4da1-96ab-1b004e5cdf9d",
"date_created": "2026-02-17T05:53:07.510000Z",
"date_modified": "2026-02-17T05:53:07.510000Z",
"file_hash": "b1b907415a3ad4c592337e3f50c202dedcb63c9bf5f4b0bf582a538f89397818",
"private": false,
"record": {
"abstract": "Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.",
"arxiv_id": "2601.06165",
"authors": [
"Dasol Choi",
"Guijin Son",
"Hanwool Lee",
"Minhyuk Kim",
"Hyunwoo Ko",
"Teabin Lim",
"Ahn Eungyeol",
"Jungwhan Kim",
"Seunghyeok Hong",
"Youngsook Song"
],
"categories": [
"cs.CV",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models",
"url": "https://arxiv.org/abs/2601.06165",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "69db0609-f08f-42ec-a687-05bf62ecdddc",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}