---
license: mit
license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B/blob/main/LICENSE
base_model: deepreinforce-ai/Ornith-1.0-9B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
  - awq
  - w4a16
  - int4
  - compressed-tensors
  - quantized
  - llmcompressor
  - vllm
  - agentic-coding
  - code
---

# Ornith-1.0-9B-AWQ-W4A16-ASYM

INT4 **AWQ (W4A16 asymmetric, group size 128)** quantization of
[deepreinforce-ai/Ornith-1.0-9B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) —
a 9B dense agentic-coding model (Qwen3.5 / Gemma 4 lineage) with a hybrid
linear-attention + full-attention backbone.

Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor). Weights are
physically packed to 4 bits (`compressed-tensors`) and load natively in **vLLM** on Ampere+
GPUs (Marlin kernels). This is a weight-only quant — activations stay BF16.

- **Size:** ~18 GB (BF16) → **8.1 GB** on disk (single `model.safetensors` shard)
- **Scheme:** `W4A16_ASYM`, group size 128
- **Calibration:** 256 samples from [`ise-uiuc/Magicoder-Evol-Instruct-110K`](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) (code-domain), max sequence length 2048, applied through the model chat template
- **Tooling:** llm-compressor 0.12, transformers 5.10, calibrated on 2×RTX 3090

## What is quantized

This is a hybrid model (mixed `linear_attention` / `full_attention` layers). The recipe
follows the proven approach for this architecture: quantize the standard projections, keep the
sensitive state-space gates and non-LLM parts in BF16.

**Quantized to INT4 (200 `Linear` modules):**
- full-attention `q_proj / k_proj / v_proj / o_proj` (8 layers)
- all MLP `gate_proj / up_proj / down_proj` (32 layers)
- linear-attention `out_proj`, `in_proj_qkv`, `in_proj_z` (24 layers)

**Kept in BF16 (`ignore`, 159 `Linear` modules):**
- the entire vision tower — `re:model\.visual\..*` (110)
- linear-attention SSM gates `in_proj_a` / `in_proj_b` (48) — sensitive to the recurrence
- `lm_head`, and any `mtp` head

## Usage (vLLM)

```bash
vllm serve Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM --trust-remote-code
```

Expected GPU memory ≈ checkpoint size + KV cache + activations.

### Recommended sampling

The model emits `<think>...</think>` reasoning blocks before the final answer.

```
temperature = 0.6, top_p = 0.95, top_k = 20
```

## Reproducibility

The `recipe.yaml` written by llm-compressor is included in this repo. `config.json` carries the
`quantization_config` (`compressed-tensors`, format `pack-quantized`) describing the exact
scheme, group size and ignore list.

## Notes

- Quantizing `lm_head` and the vision tower hurts quality for little gain, so they stay BF16.
  Together with the large embedding table and the BF16 SSM gates, this is why a 9B hybrid
  compresses to ~8 GB rather than the ~4.5 GB of a pure-transformer 7B.
- Loading this checkpoint in plain `transformers` transparently **decompresses** the weights
  back to BF16, so VRAM there looks like the original model. The real INT4 memory saving is
  realized by vLLM's native 4-bit kernels.