---
license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license
base_model: nvidia/Cosmos3-Nano
tags:
- cosmos3
- robotics
- vision-language
- quantized
- w4a16
- compressed-tensors
- vllm
pipeline_tag: image-text-to-text
---

# Cosmos3-Nano-Reasoner-W4A16 (INT4)

A 4-bit (W4A16) quantization of the **understanding tower** of
[`nvidia/Cosmos3-Nano`](https://huggingface.co/nvidia/Cosmos3-Nano) — the
Qwen3-VL-based vision-language **reasoner**. It **sees images/video + text and
outputs text** (e.g. high-level robot commands). Built to run the Cosmos 3
"brain" on a **single 12 GB consumer GPU** (RTX 4070 Ti Super class).

> Verified: loaded under a hard 12 GB cap and, given a camera frame with an
> obstacle on the right, correctly emitted `TURN LEFT`.

## What's included / excluded

Cosmos3-Nano is a Mixture-of-Transformers omni model. NVIDIA's own reasoner
loader (`transformers_cosmos3` / `vllm-cosmos3`) loads only the understanding
tower; the generation towers are dropped by design. This checkpoint therefore
contains:

- ✅ **Language reasoner** (Qwen3-VL text decoder) — quantized to INT4 W4A16.
- ✅ **Vision tower** (image/video encoder) — kept at original precision (bf16).
- ✅ All normalization / embedding tensors — kept at original precision.
- ❌ **Image/video generation, audio, and action diffusion towers** — not
  included. They are not loadable/servable by this stack and do not fit 12 GB
  at any precision. Excluding them is what makes this run on a consumer GPU.

Use this when you want the model to **look and decide / command**, not to
generate images or video.

## Quantization

- Method: `llm-compressor` `QuantizationModifier`, scheme **W4A16** (group 128,
  symmetric), data-free RTN.
- Targets: language `Linear` layers only. Vision tower, `lm_head`, and all
  norms left in original precision (`ignore` list).
- Format: `compressed-tensors` (`pack-quantized`), served via Marlin INT4
  kernels on Ampere/Ada GPUs.
- Size: ~6.8 GB on disk (vs ~30 GB bf16 reasoner+vision).

## Run it (vLLM + vllm-cosmos3)

```bash
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"

# native sampler avoids a flashinfer JIT compile that needs a CUDA toolchain
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve NathanTH141/Cosmos3-Nano-Reasoner-W4A16 \
  --hf-overrides '{"architectures":["Cosmos3ReasonerForConditionalGeneration"]}' \
  --max-model-len 4096 --gpu-memory-utilization 0.9 --port 8000
```

Then send an OpenAI-style chat request with an `image_url` (a camera frame) and
a text instruction; the model replies with your command.

## Notes / compatibility fixes baked in

- `config.json` `allow_patterns_overrides` is set to `["*.safetensors"]` (the
  base model uses a subdir layout that vLLM's flat loader can't find).
- A `model.safetensors.index.json` is included so vLLM locates the weights.
- Loading the base model in transformers requires passing the config explicitly
  (`config=AutoConfig.from_pretrained(...)`) so the vision merger builds at
  out_hidden_size 4096; this checkpoint is already correct.

## License

Derivative of `nvidia/Cosmos3-Nano`; governed by the
[NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license).
You are responsible for compliance, including any safety/guardrail requirements
that apply to your use.