dorsal/arxiv
View SchemaBeyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics
| Authors | Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma |
|---|---|
| Categories | |
| ArXiv ID | 2601.07197vv1 |
| URL | https://arxiv.org/abs/2601.07197 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Post-training activation compression is essential for deploying Large Language Models (LLMs) on resource-constrained hardware. However, standard methods like Singular Value Decomposition (SVD) are gradient-blind: they preserve high-variance dimensions regardless of their impact on factual knowledge preservation. We introduce Fisher-Aligned Subspace Compression (FASC), a knowledge-aware compression framework that selects subspaces by directly modeling activation-gradient coupling, minimizing a second-order surrogate of the loss function. FASC leverages the Fisher Information Matrix to identify dimensions critical for factual knowledge, which often reside in low-variance but high-gradient-sensitivity subspaces. We propose the Dependence Violation Score (\r{ho}) as a general-purpose diagnostic metric that quantifies activation-gradient coupling, revealing where factual knowledge is stored within transformer architectures. Extensive experiments on Mistral-7B and Llama-3-8B demonstrate that FASC preserves 6-8% more accuracy on knowledge-intensive benchmarks (MMLU, LAMA) compared to variance-based methods at 50% rank reduction, effectively enabling a 7B model to match the factual recall of a 13B uncompressed model. Our analysis reveals that \r{ho} serves as a fundamental signal of stored knowledge, with high-\r{ho} layers emerging only when models internalize factual associations during training.
{
"annotation_id": "e8d941f7-9157-4190-810d-084f97484eb2",
"date_created": "2026-02-17T05:53:11.759000Z",
"date_modified": "2026-02-17T05:53:11.759000Z",
"file_hash": "002e93ce193a764b334fa1e52570d194c183ecaf2fc622309569776ec8255dc9",
"private": false,
"record": {
"abstract": "Post-training activation compression is essential for deploying Large Language Models (LLMs) on resource-constrained hardware. However, standard methods like Singular Value Decomposition (SVD) are gradient-blind: they preserve high-variance dimensions regardless of their impact on factual knowledge preservation. We introduce Fisher-Aligned Subspace Compression (FASC), a knowledge-aware compression framework that selects subspaces by directly modeling activation-gradient coupling, minimizing a second-order surrogate of the loss function. FASC leverages the Fisher Information Matrix to identify dimensions critical for factual knowledge, which often reside in low-variance but high-gradient-sensitivity subspaces. We propose the Dependence Violation Score (\\r{ho}) as a general-purpose diagnostic metric that quantifies activation-gradient coupling, revealing where factual knowledge is stored within transformer architectures. Extensive experiments on Mistral-7B and Llama-3-8B demonstrate that FASC preserves 6-8% more accuracy on knowledge-intensive benchmarks (MMLU, LAMA) compared to variance-based methods at 50% rank reduction, effectively enabling a 7B model to match the factual recall of a 13B uncompressed model. Our analysis reveals that \\r{ho} serves as a fundamental signal of stored knowledge, with high-\\r{ho} layers emerging only when models internalize factual associations during training.",
"arxiv_id": "2601.07197",
"authors": [
"Ibne Farabi Shihab",
"Sanjeda Akter",
"Anuj Sharma"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics",
"url": "https://arxiv.org/abs/2601.07197",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "3125cb17-2597-4ffd-8f04-92178399e693",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}