Instructions to use CohereLabs/North-Micro-Vision-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CohereLabs/North-Micro-Vision-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="CohereLabs/North-Micro-Vision-Instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("CohereLabs/North-Micro-Vision-Instruct")
model = AutoModelForMultimodalLM.from_pretrained("CohereLabs/North-Micro-Vision-Instruct", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CohereLabs/North-Micro-Vision-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CohereLabs/North-Micro-Vision-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Micro-Vision-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/CohereLabs/North-Micro-Vision-Instruct

SGLang

How to use CohereLabs/North-Micro-Vision-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CohereLabs/North-Micro-Vision-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Micro-Vision-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CohereLabs/North-Micro-Vision-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Micro-Vision-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use CohereLabs/North-Micro-Vision-Instruct with Docker Model Runner:
```
docker model run hf.co/CohereLabs/North-Micro-Vision-Instruct
```

d-rau commited on 2 days ago

Commit

373bda9

verified ·

1 Parent(s): 0add1b1

Update README.md

Browse files

Files changed (1) hide show

README.md +23 -0

README.md CHANGED Viewed

@@ -151,6 +151,16 @@ print(response)
 The example uses the recommended Transformers sampling settings. For deterministic output, set `do_sample=False` and omit `temperature`, `top_p`, and `top_k`.
 ### Grounding Coordinates
 Bounding boxes are returned as `[x1, y1, x2, y2]` on a normalized 0–1000 scale. Map them back to the original image by scaling each axis:
@@ -194,6 +204,19 @@ North Micro Vision Instruct is intended for research and development use cases s
 - Multimodal training used an 8K-token context; longer contexts have not been validated.
 - Native-resolution inputs can increase memory use and latency as image dimensions grow.
 ## Benchmark Results
 The complete comparison is provided below. We ran vision-language and text-only evaluations with [VLMEvalKit](https://github.com/open-compass/vlmevalkit), capping generation at 1,024 tokens; see the [technical blog post](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) for the full methodology.

 The example uses the recommended Transformers sampling settings. For deterministic output, set `do_sample=False` and omit `temperature`, `top_p`, and `top_k`.
+## Architecture
+North Micro Vision combines a custom-trained 400M-parameter native-resolution vision encoder with an in-house 2B-parameter language model North Micro LLM. The language model follows our Command A+ architecture, interleaving three sliding-window attention layers that use rotary positional embeddings with one global attention layer without positional embeddings. The vision encoder combines 2D RoPE with learned 1D positional embeddings to preserve spatial structure across native-resolution inputs.
+The projector maps visual features into the language model's embedding space. Following DeepStack, patch embeddings from multiple vision-encoder layers are injected into corresponding early LLM layers, giving the language model access to visual representations at different levels of abstraction.
+![North-Micro-Vision-Instruct-Architecture](https://cdn-uploads.huggingface.co/production/uploads/66d732effe6684fc16b12c28/hPsTh7FX3ONJXN3WfE0BX.png)
+*High-level North Micro Vision architecture, consisting of a native-resolution vision encoder, a projector, and a language model.*
 ### Grounding Coordinates
 Bounding boxes are returned as `[x1, y1, x2, y2]` on a normalized 0–1000 scale. Map them back to the original image by scaling each axis:
 - Multimodal training used an 8K-token context; longer contexts have not been validated.
 - Native-resolution inputs can increase memory use and latency as image dimensions grow.
+## Ecosystem Support
+### Fast Inference 🚀
+- [MLX-VLM model weights](https://huggingface.co/collections/mlx-community/north-micro-vision) - Community-contributed by Prince Canuma and Neywa.
+### Fine-tuning
+In partnership with NVIDIA, we're also shipping an AutoModel recipe for North Micro Vision, so developers can fine-tune and deploy it on NVIDIA GPUs right out of the box.
+- [NVIDIA AutoModel recipe](https://github.com/NVIDIA-NeMo/Automodel/tree/main/examples/vlm_finetune/cohere_micro_vision) - Fine-tune and deploy North Micro Vision on NVIDIA GPUs.
+- [Axolotl fine-tuning support](https://docs.axolotl.ai/docs/models/cohere-north-micro-vision-instruct.html) - Community-supported fine-tuning using the Axolotl framework.
 ## Benchmark Results
 The complete comparison is provided below. We ran vision-language and text-only evaluations with [VLMEvalKit](https://github.com/open-compass/vlmevalkit), capping generation at 1,024 tokens; see the [technical blog post](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) for the full methodology.