dorsal/arxiv
View SchemaRepresentations of Text and Images Align From Layer One
| Authors | Evžen Wybitul, Javier Rando, Florian Tramèr, Stanislav Fort |
|---|---|
| Categories | |
| ArXiv ID | 2601.08017vv1 |
| URL | https://arxiv.org/abs/2601.08017 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as "Jupiter", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model's representation space by backtracing through its image processing components.
{
"annotation_id": "ff0f2bf2-a5de-4b05-a10e-4fe5a22ae267",
"date_created": "2026-02-17T05:53:15.041000Z",
"date_modified": "2026-02-17T05:53:15.041000Z",
"file_hash": "8e99bc1c934647dc95ff7b271f9bc351cb76184a52c33485209fbe2ea3c92284",
"private": false,
"record": {
"abstract": "We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as \"Jupiter\", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model\u0027s representation space by backtracing through its image processing components.",
"arxiv_id": "2601.08017",
"authors": [
"Ev\u017een Wybitul",
"Javier Rando",
"Florian Tram\u00e8r",
"Stanislav Fort"
],
"categories": [
"cs.CV",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Representations of Text and Images Align From Layer One",
"url": "https://arxiv.org/abs/2601.08017",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "14f9852e-e4bc-4ff3-a563-b708a4d84c32",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}