---
license: apache-2.0
base_model: qvac/VisionPsy-Nano-460M
tags:
- mlx
- vlm
- vision-language-model
- apple-silicon
- siglip2
- smollm2
language:
- en
- zh
library_name: mlx
pipeline_tag: image-text-to-text
---

# VisionPsy-Nano-460M-MLX

MLX port of [**qvac/VisionPsy-Nano-460M**](https://huggingface.co/qvac/VisionPsy-Nano-460M), a compact 460M-parameter vision-language model from Tether AI Research, converted to run natively on Apple Silicon.

- **Architecture:** SigLIP2-base-patch16-512 vision encoder + pixel-shuffle modality projector + SmolLM2-360M-Instruct decoder
- **Parameters:** ~460M
- **Precision:** bfloat16 (~1.0 GB on disk, down from 2.0 GB fp32)
- **Runtime:** MLX on Apple Silicon (M-series)
- **License:** Apache-2.0

## Benchmarks (MLX bf16, M-series)

Measured across 7 images x 5 prompts, 64 max new tokens, greedy decode:

| Metric | Standard | Flash |
|---|---|---|
| Avg decode tok/s | 99 | 152 |
| Median decode tok/s | 90 | 157 |
| Avg peak GPU memory | 2.64 GB | 2.64 GB |
| Load time | ~0.4 s | ~0.7 s |

Per-prompt-type medians (Standard):

| Prompt type | Median tok/s | Example |
|---|---|---|
| Describe (EN, 1 sentence) | 158.7 | "A smiling man in a white lab coat gestures with his right hand..." |
| What text appears? | 90.3 | "OICOMELVANG" |
| Count objects/people | 40.0 | "There are 3 people in the image." |
| Main subject | 59.2 | "The main subject is a man wearing a white lab coat." |
| Describe (ZH) | 130.4 | "他說：\"OICOMELVANG, 25158\"" |

Full 70-run matrix (Standard + Flash) is at [github.com/KaedeTai/mlx-video/tree/visionpsy-mlx-port](https://github.com/KaedeTai/mlx-video/tree/visionpsy-mlx-port).

## Usage

This repo uses the mlx-vlm-style layout (`text_config` + `vision_config` top-level, `language_model.*` / `vision_tower.*` / `multi_modal_projector.*` tensor prefixes) so it slots cleanly into `mlx-vlm` once a `visionpsy_nano` handler lands there. Until then, load it via the MLX port bundled in **mlx-video** (branch `visionpsy-mlx-port`):

```bash
pip install mlx safetensors transformers pillow
git clone -b visionpsy-mlx-port https://github.com/KaedeTai/mlx-video.git
cd mlx-video
```

```python
from huggingface_hub import snapshot_download
from mlx_video.models.visionpsy_nano import load_visionpsy_nano
from mlx_video.models.visionpsy_nano.processor import load_processor
from PIL import Image

# Snapshot from HF (or point at your local folder)
path = snapshot_download("KaedeTai/VisionPsy-Nano-460M-MLX")

model, cfg = load_visionpsy_nano(path)
proc = load_processor(path, cfg=cfg)

img = Image.open("photo.jpg").convert("RGB")
batch = proc("Describe this image in one sentence.", image=img)

tokens = list(model.generate(
    batch["input_ids"],
    pixel_values=batch["pixel_values"],
    image_token_id=batch["image_token_id"],
    max_new_tokens=64,
    eos_token_id=proc.tokenizer.eos_token_id,
))
print(proc.decode(tokens, skip_special_tokens=True))
```

*Note:* the port's `load_visionpsy_nano` reads the repacked config via a compat shim; the original `_original_config` block is retained inside `config.json` for round-tripping.

## What changed vs the original

- **fp32 -> bf16.** Weights cast to bfloat16. Outputs verified byte-identical on greedy decode for both variants.
- **Prefix rename.** Tensor names moved from `decoder.*` / `vision_encoder.*` / `MP.*` to `language_model.*` / `vision_tower.*` / `multi_modal_projector.*` to match mlx-vlm conventions.
- **Config reshape.** Flat `lm_*` / `vit_*` keys refactored into nested `text_config` / `vision_config` blocks with standard field names (`hidden_size`, `num_hidden_layers`, etc.).
- **Stale buffers dropped.** `decoder.rotary_embd.*` buffers removed — MLX's `nn.RoPE` computes frequencies on the fly.

## Attribution

- **Original model:** Tether AI Research / QVAC — [qvac/VisionPsy-Nano-460M](https://huggingface.co/qvac/VisionPsy-Nano-460M) (Apache-2.0)
- **MLX port + weight repack:** [KaedeTai](https://huggingface.co/KaedeTai)
- **Base components:** SigLIP2 (Google), SmolLM2 (Hugging Face)

If you use these weights, please also cite the original QVAC release and the base model authors.

## See also

- Flash variant: [KaedeTai/VisionPsy-Nano-460M-Flash-MLX](https://huggingface.co/KaedeTai/VisionPsy-Nano-460M-Flash-MLX)
- Original release blog + benchmarks: [qvac/VisionPsy-Nano-460M](https://huggingface.co/qvac/VisionPsy-Nano-460M)
- MLX port source: [github.com/KaedeTai/mlx-video @ visionpsy-mlx-port](https://github.com/KaedeTai/mlx-video/tree/visionpsy-mlx-port)