README.md

---
library_name: transformers
pipeline_tag: feature-extraction
tags:
  - encoder-only
  - multimodal
  - image-text-retrieval
  - image-text-matching
  - siglip2
  - lfm2.5
  - gptq
---

# LFM2.5 Encoder 230M + SigLIP2

A public GPTQ INT4 release of an encoder-only image-text model built by augmenting
Liquid AI's LFM2.5 Encoder 230M with the `google/siglip2-base-patch16-256` vision
encoder and a learned 32-token visual projector.

> This is an external vision augmentation of LFM2.5. The original LFM2.5 checkpoint is
> not claimed to be natively multimodal.

## What it does

The model produces image-text representations and image-text matching scores for:

- Image-to-text retrieval
- Text-to-image retrieval
- Image-text matching
- Representation extraction
- Encoder-side masked-token experiments

It is **not** an autoregressive model and does not generate captions or chat responses.

## Architecture

```text
Image
  ↓
SigLIP2 base patch16 256
  [batch, 256, 768]
  ↓
Learned visual projector / 32-token adapter
  [batch, 32, 1024]
  ↓
LFM2.5 bidirectional encoder
  [batch, sequence, 1024]
  ↓
Image-text embedding or matching score
```

The image branch uses BOS, an image marker, and projected visual tokens. During clean
retrieval evaluation, the candidate caption is never provided to the image branch.

## Quick start

Clone the repository and install the tested runtime dependencies:

```bash
git clone https://huggingface.co/konic-labs/LFM2.5-multimodal-encoder-230M
cd LFM2.5-multimodal-encoder-230M

pip install torch transformers pillow accelerate safetensors compressed-tensors
```

GPTQ loading requires a `compressed-tensors` installation compatible with the model
configuration. The tested quantization details are recorded in `recipe.yaml` and
`compression_manifest.json`.

The SigLIP2 vision model is downloaded from Hugging Face on first use. Run one image-text
matching example:

```bash
python run_multimodal.py \
  --model . \
  --image /path/to/image.jpg \
  --text "A photograph of a small red vehicle beside a building." \
  --device cuda:0 \
  --dtype float32
```

The runner reports the visual feature shape, projected token shape, fused hidden-state
shape, matching score, and finite-output status.

Expected shapes include:

```text
visual_features: [1, 256, 768]
image_tokens:    [1, 32, 1024]
hidden_state:    [1, sequence, 1024]
```

The task adapter contains the matching head and the projector state used by the bundled
runner. `projector.pt` is also provided as a standalone projector artifact for modular
experiments.

## Main files

| File | Purpose |
|---|---|
| `model.safetensors` | GPTQ INT4 LFM2.5 encoder weights |
| `config.json` | Transformers model configuration |
| `modeling_lfm2_bidirectional.py` | Bidirectional LFM2 runtime implementation |
| `multimodal_task_adapter.pt` | Projector and image-text matching head |
| `projector.pt` | Standalone SigLIP2-to-LFM projector |
| `multimodal_reference_manifest.json` | Vision and adapter configuration |
| `compression_manifest.json` | GPTQ method and calibration provenance |
| `recipe.yaml` | Quantization recipe |
| `run_multimodal.py` | Local image-text inference example |
| `evaluation/` | Saved metrics and deep-evaluation summaries |

## Evaluation

All clean evaluations use an image-only representation for retrieval. Retrieval metrics
were computed on 12,500 held-out MONET pairs with 200 bootstrap resamples for confidence
intervals.

### MONET test set

| Direction | R@1 | R@5 | R@10 | Mean rank |
|---|---:|---:|---:|---:|
| Image → text | 0.1091 | 0.2959 | 0.4090 | 63.38 |
| Text → image | 0.1460 | 0.3418 | 0.4590 | 53.78 |

### External CC3M subset

A bounded, 2,000-example CC3M subset was used for external evaluation:

| Direction | R@1 | R@5 | R@10 | Mean rank |
|---|---:|---:|---:|---:|
| Image → text | 0.0865 | 0.2190 | 0.3060 | 137.34 |
| Text → image | 0.1145 | 0.2835 | 0.3865 | 112.76 |

### Visual ablation

On 512 MONET examples:

| Visual input | Image → text R@1 | Text → image R@1 |
|---|---:|---:|
| Trained visual tokens | 0.4883 | 0.5352 |
| Zero visual tokens | 0.0020 | 0.0000 |
| Random visual tokens | 0.0020 | 0.0000 |
| Shuffled visual tokens | 0.0117 | 0.0215 |

These controls show that the learned visual tokens materially affect the image
representation.

### Matching

Cyclic-negative matching AUROC is `0.9733` on MONET. Text-nearest hard-negative AUROC
is `0.4961`, so the current model is substantially better at coarse compatibility than
fine-grained semantic discrimination.

## Compression details and size comparison

The released LFM2.5 body uses GPTQ INT4 W4A16 compression:

| Compression setting | Value |
|---|---|
| Method | GPTQ INT4 |
| Weight/activation format | W4A16 |
| Quantized layers | Linear layers; `lm_head` excluded |
| Block size | 128 |
| Activation order | Static |
| Dampening fraction | 0.01 |
| Calibration | 256 fused multimodal `inputs_embeds` samples |
| Maximum calibration sequence length | 160 tokens |
| Compressor revision | `8cec0acc1931de6f8f73257151ab7007c14dbf4e` |

The calibration samples contained actual projected visual tokens rather than text-only
inputs, so the quantization pass reflects the multimodal path.

### Storage comparison

| Artifact | Model weights | Full package | Relative to original package |
|---|---:|---:|---:|
| Original LFM2.5 Encoder 230M FP32 | 918.79 MB | 923.65 MB | 100% |
| Clean BF16 reference | 459.40 MB | 474.81 MB | 51.4% |
| Clean GPTQ INT4 release | 355.03 MB | 370.46 MB | 40.1% |

The GPTQ release is approximately **59.9% smaller than the original package** and
**22.0% smaller than the clean BF16 package**. The external SigLIP2 vision tower is not
included in these sizes and is downloaded separately.

### Fidelity and runtime

| Metric | GPTQ INT4 |
|---|---:|
| Hidden cosine versus BF16 | 0.9427 |
| Logit cosine versus BF16 | 0.9654 |
| Multimodal latency, batch 4 | 45.7 ms on NVIDIA L4 |
| Peak multimodal VRAM | 2.38 GiB on NVIDIA L4 |

The current GPTQ backend is intended primarily for weight-size reduction. Native BF16
is faster and uses less runtime VRAM in the tested environment.

## Limitations

- Encoder-only: no generation is supported.
- Hard-negative matching remains near chance.
- Masked-token prediction does not currently improve over text-only or zero-image
  controls.
- External evaluation is a bounded CC3M subset rather than a full benchmark.
- The SigLIP2 vision tower remains an external Hugging Face dependency.

## Reproducibility

The `evaluation/` directory contains the saved BF16, GPTQ, and external CC3M metrics,
retrieval summary, matching results, ablations, robustness results, and the detailed
deep-evaluation summary.

The compact release corresponds to the clean image-only training branch and should be
used instead of the earlier paired-caption artifact.