dorsal/arxiv
View SchemaCLIMP: Contrastive Language-Image Mamba Pretraining
| Authors | Nimrod Shabtay, Itamar Zimerman, Eli Schwartz, Raja Giryes |
|---|---|
| Categories | |
| ArXiv ID | 2601.06891vv1 |
| URL | https://arxiv.org/abs/2601.06891 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.
{
"annotation_id": "26ff10ed-2506-483a-a5dd-07843c39e737",
"date_created": "2026-02-17T05:53:08.652000Z",
"date_modified": "2026-02-17T05:53:08.652000Z",
"file_hash": "d29a00f6b34b0822c753cf9c9b947eed4f64a0ffdb978039b5cd592a942e0096",
"private": false,
"record": {
"abstract": "Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI\u0027s CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP\u0027s fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.",
"arxiv_id": "2601.06891",
"authors": [
"Nimrod Shabtay",
"Itamar Zimerman",
"Eli Schwartz",
"Raja Giryes"
],
"categories": [
"cs.CV"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "CLIMP: Contrastive Language-Image Mamba Pretraining",
"url": "https://arxiv.org/abs/2601.06891",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "692dc3b2-46f3-457a-b7fa-bbf7235c663b",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}