Upload folder using huggingface_hub

9e23e2a verified about 2 months ago

3.6 kB

license: other
license_name: nvidia-open-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license
base_model: nvidia/Cosmos3-Nano
tags:
  - cosmos3
  - robotics
  - vision-language
  - quantized
  - w4a16
  - compressed-tensors
  - vllm
pipeline_tag: image-text-to-text

Cosmos3-Nano-Reasoner-W4A16 (INT4)

A 4-bit (W4A16) quantization of the understanding tower of nvidia/Cosmos3-Nano — the Qwen3-VL-based vision-language reasoner. It sees images/video + text and outputs text (e.g. high-level robot commands). Built to run the Cosmos 3 "brain" on a single 12 GB consumer GPU (RTX 4070 Ti Super class).

Verified: loaded under a hard 12 GB cap and, given a camera frame with an obstacle on the right, correctly emitted TURN LEFT.

What's included / excluded

Cosmos3-Nano is a Mixture-of-Transformers omni model. NVIDIA's own reasoner loader (transformers_cosmos3 / vllm-cosmos3) loads only the understanding tower; the generation towers are dropped by design. This checkpoint therefore contains:

✅ Language reasoner (Qwen3-VL text decoder) — quantized to INT4 W4A16.
✅ Vision tower (image/video encoder) — kept at original precision (bf16).
✅ All normalization / embedding tensors — kept at original precision.
❌ Image/video generation, audio, and action diffusion towers — not included. They are not loadable/servable by this stack and do not fit 12 GB at any precision. Excluding them is what makes this run on a consumer GPU.

Use this when you want the model to look and decide / command, not to generate images or video.

Quantization

Method: llm-compressor QuantizationModifier, scheme W4A16 (group 128, symmetric), data-free RTN.
Targets: language Linear layers only. Vision tower, lm_head, and all norms left in original precision (ignore list).
Format: compressed-tensors (pack-quantized), served via Marlin INT4 kernels on Ampere/Ada GPUs.
Size: ~6.8 GB on disk (vs ~30 GB bf16 reasoner+vision).

Run it (vLLM + vllm-cosmos3)

uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"

# native sampler avoids a flashinfer JIT compile that needs a CUDA toolchain
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve NathanTH141/Cosmos3-Nano-Reasoner-W4A16 \
  --hf-overrides '{"architectures":["Cosmos3ReasonerForConditionalGeneration"]}' \
  --max-model-len 4096 --gpu-memory-utilization 0.9 --port 8000

Then send an OpenAI-style chat request with an image_url (a camera frame) and a text instruction; the model replies with your command.

Notes / compatibility fixes baked in

config.json allow_patterns_overrides is set to ["*.safetensors"] (the base model uses a subdir layout that vLLM's flat loader can't find).
A model.safetensors.index.json is included so vLLM locates the weights.
Loading the base model in transformers requires passing the config explicitly (config=AutoConfig.from_pretrained(...)) so the vision merger builds at out_hidden_size 4096; this checkpoint is already correct.

License

Derivative of nvidia/Cosmos3-Nano; governed by the NVIDIA Open Model License. You are responsible for compliance, including any safety/guardrail requirements that apply to your use.