dorsal/arxiv
View SchemaMTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
| Authors | Zheyuan Liu, Dongwhi Kim, Yixin Wan, Xiangchi Yuan, Zhaoxuan Tan, Fengran Mo, Meng Jiang |
|---|---|
| Categories | |
| ArXiv ID | 2601.06757vv1 |
| URL | https://arxiv.org/abs/2601.06757 |
| License | http://creativecommons.org/publicdomain/zero/1.0/ |
Abstract
Multimodal large language models (MLLMs) are increasingly deployed as assistants that interact through text and images, making it crucial to evaluate contextual safety when risk depends on both the visual scene and the evolving dialogue. Existing contextual safety benchmarks are mostly single-turn and often miss how malicious intent can emerge gradually or how the same scene can support both benign and exploitative goals. We introduce the Multi-Turn Multimodal Contextual Safety Benchmark (MTMCS-Bench), a benchmark of realistic images and multi-turn conversations that evaluates contextual safety in MLLMs under two complementary settings, escalation-based risk and context-switch risk. MTMCS-Bench offers paired safe and unsafe dialogues with structured evaluation. It contains over 30 thousand multimodal (image+text) and unimodal (text-only) samples, with metrics that separately measure contextual intent recognition, safety-awareness on unsafe cases, and helpfulness on benign ones. Across eight open-source and seven proprietary MLLMs, we observe persistent trade-offs between contextual safety and utility, with models tending to either miss gradual risks or over-refuse benign dialogues. Finally, we evaluate five current guardrails and find that they mitigate some failures but do not fully resolve multi-turn contextual risks.
{
"annotation_id": "55d83332-0275-4c60-b31b-a520fe0e5270",
"date_created": "2026-02-17T05:53:08.613000Z",
"date_modified": "2026-02-17T05:53:08.613000Z",
"file_hash": "14a354c4025fa7d924e72f5f6d0f9279768b79ee6e6dbe2a9a296546ae0c90ee",
"private": false,
"record": {
"abstract": "Multimodal large language models (MLLMs) are increasingly deployed as assistants that interact through text and images, making it crucial to evaluate contextual safety when risk depends on both the visual scene and the evolving dialogue. Existing contextual safety benchmarks are mostly single-turn and often miss how malicious intent can emerge gradually or how the same scene can support both benign and exploitative goals. We introduce the Multi-Turn Multimodal Contextual Safety Benchmark (MTMCS-Bench), a benchmark of realistic images and multi-turn conversations that evaluates contextual safety in MLLMs under two complementary settings, escalation-based risk and context-switch risk. MTMCS-Bench offers paired safe and unsafe dialogues with structured evaluation. It contains over 30 thousand multimodal (image+text) and unimodal (text-only) samples, with metrics that separately measure contextual intent recognition, safety-awareness on unsafe cases, and helpfulness on benign ones. Across eight open-source and seven proprietary MLLMs, we observe persistent trade-offs between contextual safety and utility, with models tending to either miss gradual risks or over-refuse benign dialogues. Finally, we evaluate five current guardrails and find that they mitigate some failures but do not fully resolve multi-turn contextual risks.",
"arxiv_id": "2601.06757",
"authors": [
"Zheyuan Liu",
"Dongwhi Kim",
"Yixin Wan",
"Xiangchi Yuan",
"Zhaoxuan Tan",
"Fengran Mo",
"Meng Jiang"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://creativecommons.org/publicdomain/zero/1.0/",
"title": "MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues",
"url": "https://arxiv.org/abs/2601.06757",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "ea3d69a4-98be-450c-a6a4-d69c5e725a32",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}