dorsal/arxiv
View SchemaFACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG
| Authors | Maxime Dassen, Rebecca Kotula, Kenton Murray, Andrew Yates, Dawn Lawrie, Efsun Kayi, James Mayfield, Kevin Duh |
|---|---|
| Categories | |
| ArXiv ID | 2601.05866vv2 |
| URL | https://arxiv.org/abs/2601.05866 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Retrieval-Augmented Generation (RAG) models are critically undermined by citation hallucinations, a deceptive failure where a model confidently cites a source that fails to support its claim. Existing work often attributes hallucination to a simple over-reliance on the model's parametric knowledge. We challenge this view and introduce FACTUM (Framework for Attesting Citation Trustworthiness via Underlying Mechanisms), a framework of four mechanistic scores measuring the distinct contributions of a model's attention and FFN pathways, and the alignment between them. Our analysis reveals two consistent signatures of correct citation: a significantly stronger contribution from the model's parametric knowledge and greater use of the attention sink for information synthesis. Crucially, we find the signature of a correct citation is not static but evolves with model scale. For example, the signature of a correct citation for the Llama-3.2-3B model is marked by higher pathway alignment, whereas for the Llama-3.1-8B model, it is characterized by lower alignment, where pathways contribute more distinct, orthogonal information. By capturing this complex, evolving signature, FACTUM outperforms state-of-the-art baselines by up to 37.5% in AUC. Our findings reframe citation hallucination as a complex, scale-dependent interplay between internal mechanisms, paving the way for more nuanced and reliable RAG systems.
{
"annotation_id": "e8e75aca-031f-482e-b6b1-f5399f4c005a",
"date_created": "2026-02-17T05:53:04.820000Z",
"date_modified": "2026-02-17T05:53:04.820000Z",
"file_hash": "1f68b32daabfaa37d76da28a931ece454db5034932f32d1b994faa2a65758bc2",
"private": false,
"record": {
"abstract": "Retrieval-Augmented Generation (RAG) models are critically undermined by citation hallucinations, a deceptive failure where a model confidently cites a source that fails to support its claim. Existing work often attributes hallucination to a simple over-reliance on the model\u0027s parametric knowledge. We challenge this view and introduce FACTUM (Framework for Attesting Citation Trustworthiness via Underlying Mechanisms), a framework of four mechanistic scores measuring the distinct contributions of a model\u0027s attention and FFN pathways, and the alignment between them. Our analysis reveals two consistent signatures of correct citation: a significantly stronger contribution from the model\u0027s parametric knowledge and greater use of the attention sink for information synthesis. Crucially, we find the signature of a correct citation is not static but evolves with model scale. For example, the signature of a correct citation for the Llama-3.2-3B model is marked by higher pathway alignment, whereas for the Llama-3.1-8B model, it is characterized by lower alignment, where pathways contribute more distinct, orthogonal information. By capturing this complex, evolving signature, FACTUM outperforms state-of-the-art baselines by up to 37.5% in AUC. Our findings reframe citation hallucination as a complex, scale-dependent interplay between internal mechanisms, paving the way for more nuanced and reliable RAG systems.",
"arxiv_id": "2601.05866",
"authors": [
"Maxime Dassen",
"Rebecca Kotula",
"Kenton Murray",
"Andrew Yates",
"Dawn Lawrie",
"Efsun Kayi",
"James Mayfield",
"Kevin Duh"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG",
"url": "https://arxiv.org/abs/2601.05866",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "16ab64b2-b08a-4bf9-9ea9-93ffa8256101",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}