Instructions to use Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM")
model = AutoModelForMultimodalLM.from_pretrained("Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM

SGLang

How to use Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM with Docker Model Runner:
```
docker model run hf.co/Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM
```

Ornith-1.0-9B-AWQ-W4A16-ASYM

INT4 AWQ (W4A16 asymmetric, group size 128) quantization of deepreinforce-ai/Ornith-1.0-9B — a 9B dense agentic-coding model (Qwen3.5 / Gemma 4 lineage) with a hybrid linear-attention + full-attention backbone.

Produced with llm-compressor. Weights are physically packed to 4 bits (compressed-tensors) and load natively in vLLM on Ampere+ GPUs (Marlin kernels). This is a weight-only quant — activations stay BF16.

Size: ~18 GB (BF16) → 8.1 GB on disk (single model.safetensors shard)
Scheme: W4A16_ASYM, group size 128
Calibration: 256 samples from ise-uiuc/Magicoder-Evol-Instruct-110K (code-domain), max sequence length 2048, applied through the model chat template
Tooling: llm-compressor 0.12, transformers 5.10, calibrated on 2×RTX 3090

What is quantized

This is a hybrid model (mixed linear_attention / full_attention layers). The recipe follows the proven approach for this architecture: quantize the standard projections, keep the sensitive state-space gates and non-LLM parts in BF16.

Quantized to INT4 (200 Linear modules):

full-attention q_proj / k_proj / v_proj / o_proj (8 layers)
all MLP gate_proj / up_proj / down_proj (32 layers)
linear-attention out_proj, in_proj_qkv, in_proj_z (24 layers)

Kept in BF16 (ignore, 159 Linear modules):

the entire vision tower — re:model\.visual\..* (110)
linear-attention SSM gates in_proj_a / in_proj_b (48) — sensitive to the recurrence
lm_head, and any mtp head

Usage (vLLM)

vllm serve Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM --trust-remote-code

Expected GPU memory ≈ checkpoint size + KV cache + activations.

Recommended sampling

The model emits <think>...</think> reasoning blocks before the final answer.

temperature = 0.6, top_p = 0.95, top_k = 20

Reproducibility

The recipe.yaml written by llm-compressor is included in this repo. config.json carries the quantization_config (compressed-tensors, format pack-quantized) describing the exact scheme, group size and ignore list.

Notes

Quantizing lm_head and the vision tower hurts quality for little gain, so they stay BF16. Together with the large embedding table and the BF16 SSM gates, this is why a 9B hybrid compresses to ~8 GB rather than the ~4.5 GB of a pure-transformer 7B.
Loading this checkpoint in plain transformers transparently decompresses the weights back to BF16, so VRAM there looks like the original model. The real INT4 memory saving is realized by vLLM's native 4-bit kernels.

Downloads last month: 195

Safetensors

Model size

10B params

Tensor type

I64

I32

BF16

Model tree for Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM

Base model

ornith-ai/Ornith-1.0-9B

Quantized

(100)

this model

Collection including Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM

Quantized

Collection

Quantized models by me • 5 items • Updated 7 days ago