---
license: mit
pipeline_tag: image-text-to-text
library_name: llama.cpp
base_model:
  - sakamakismile/Ornith-1.0-35B-NVFP4
  - deepreinforce-ai/Ornith-1.0-35B
base_model_relation: quantized
language:
  - en
tags:
  - gguf
  - llama-cpp
  - qwen
  - qwen3_5_moe
  - qwen35moe
  - nvfp4
  - compressed-tensors
  - vision
  - vlm
  - multimodal
  - mixture-of-experts
  - agentic-coding
  - quantized
  - rtx-5090
  - blackwell
---

# Ornith-1.0-35B-NVFP4-noMTP-BF16-GGUF

GGUF conversion of [`sakamakismile/Ornith-1.0-35B-NVFP4`](https://huggingface.co/sakamakismile/Ornith-1.0-35B-NVFP4),
**preserving the compressed-tensors NVFP4 weights**, with a **BF16 vision projector (mmproj)** and
**BF16 (lossless) embedding & output layers**. Text-only checkpoint (no MTP head). Benchmarked on an RTX 5090 (Blackwell) with `llama-benchy`.

## Highlights

- **NVFP4 preserved** — 430 NVFP4 tensors (MoE experts, attention, linear-attention/SSM) kept in their native 4-bit format, not re-quantized.
- **Lossless embeddings & output** — `token_embd` and `lm_head` are kept at **BF16** (no Q4_K/Q6_K), so vocabulary and output-projection precision is unchanged from the source.
- **Vision / mmproj** — a separate BF16 vision projector enables image-text-to-text.
- **No MTP** — the source checkpoint ships no multi-token-prediction head, so this is a clean autoregressive (AR) model (`block_count = 40`).
- **Blackwell native FP4** — prefill/decoding use native FP4 tensor-core kernels on RTX 50-series (`sm_120`).

## Files

| File | Size | Description |
|---|---:|---|
| `Ornith-1.0-35B-NVFP4-noMTP-BF16.gguf` | 19.60 GiB | Main GGUF — NVFP4 body preserved, BF16 `token_embd`/`lm_head` |
| `mmproj-Ornith-1.0-35B-BF16.gguf` | 861 MiB | Vision projector (mmproj) for image input |

## Usage (llama.cpp)

```bash
# Text-only
llama-server -m Ornith-1.0-35B-NVFP4-noMTP-BF16.gguf --host 0.0.0.0 --port 8080 --jinja

# With vision (image-text-to-text)
llama-server -m Ornith-1.0-35B-NVFP4-noMTP-BF16.gguf \
  --mmproj mmproj-Ornith-1.0-35B-BF16.gguf \
  --host 0.0.0.0 --port 8080 --jinja
```

For a Blackwell (RTX 50xx) build with native FP4 acceleration, compile llama.cpp with CUDA 12.8+ and
`-DCMAKE_CUDA_ARCHITECTURES=120`. `-ngl 999` offloads all layers; `-c 262144` uses the full context.

This is a **thinking** model. Reasoning is emitted inside `<think>…</think>`. To disable thinking for a
turn, pass `"chat_template_kwargs": {"enable_thinking": false}` in the request body.

## Vision

The model is multimodal (image-text-to-text). Image input requires the separate `mmproj` file via `--mmproj`.
Verified on llama.cpp with a synthetic image (the model correctly described the shape and color).
Loading the mmproj has **negligible impact on text-generation throughput** (see benchmarks).

## Benchmarks

RTX 5090 (32 GB), llama.cpp **b9812** (`0e53b82a9`, MSVC 19.50, `sm_120a`, `BLACKWELL_NATIVE_FP4=1`),
`llama-benchy` (pp = 512/4096, tg = 512, runs = 3, `--latency-mode generation`),
tokenizer `deepreinforce-ai/Ornith-1.0-35B`, ctx = 262144, KV cache `q8_0`. Autoregressive (no spec decoding).

| Mode | pp | Prefill (pp) tok/s | Generation (tg) tok/s | Peak tg |
|---|---:|---:|---:|---:|
| Text | 512 | 6539.84 ± 84.57 | 193.58 ± 3.20 | 194.00 |
| Text | 4096 | 7754.81 ± 288.02 | 193.75 ± 1.92 | 194.33 |
| Vision (+mmproj) | 512 | 6076.30 ± 335.59 | 188.55 ± 1.97 | 189.00 |
| Vision (+mmproj) | 4096 | 7674.93 ± 86.51 | 190.58 ± 0.24 | 191.00 |

Coherence test PASSED in all runs; generation latency ≈ 112–125 ms.

## Runtime configuration

Currently loaded with `llama-server` on RTX 5090:

| Parameter | Value |
|---|---:|
| Context (`c`) | 196 608 |
| Max tokens | 32 768 |
| Speculative decoding | none |
| Vision projector | `mmproj-Ornith-1.0-35B-BF16.gguf` |
| KV cache type | q8_0 |
| **VRAM usage** | **22.68 GiB** |

## Quantization / conversion notes

Converted with llama.cpp's `convert_hf_to_gguf.py` directly from the `sakamakismile` NVFP4 checkpoint
(`compressed-tensors`, `format: nvfp4-pack-quantized`), which is auto-detected and re-packed into
`GGML_TYPE_NVFP4` — no de-quantization to full precision.

- **`--outtype bf16` (not Q4_K).** With `bf16`, the NVFP4 body is preserved and the large 2-D weights
  (`token_embd`, `lm_head`) stay at **BF16**; only small 1-D tensors (norms, biases, router gate, scales)
  are F32. There is **no wasteful F32 blow-up of the big weights**, so a follow-up `llama-quantize` pass is
  not required to reach a reasonable size (19.60 GiB).
- **`--no-mtp`.** The checkpoint's `config.json` declares `mtp_num_hidden_layers: 1` but ships **no MTP
  tensors**; `--no-mtp` keeps `block_count = 40` and avoids requesting an absent MTP head.
- **Tensor make-up:** NVFP4 × 430, F32 × 1161 (scales / norms / router gate), BF16 × 2 (`token_embd`, `lm_head`).

**Speed/quality trade-off of BF16 output:** keeping `lm_head` at BF16 (~1.0 GiB) costs roughly **14 % of
generation throughput** versus a fully quantized build (this model ≈ 193 tok/s vs Q4_K_M ≈ 223 tok/s on the
same hardware), because the output projection over the ~248k-token vocabulary is read every decoding step.
This build **prioritizes lossless embeddings/output**; for speed, Q6_K quantization of the output layer via `llama-quantize` is required.

## Model details

- **Original (quantized) model:** [`sakamakismile/Ornith-1.0-35B-NVFP4`](https://huggingface.co/sakamakismile/Ornith-1.0-35B-NVFP4)
- **Base model:** [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)
- **Architecture:** `qwen35moe` (Qwen3.5-MoE family; hybrid full + gated-linear attention), `block_count = 40`, no MTP
- **MoE:** 256 experts, 8 active; embedding dim 2048; context length 262144
- **Quantization:** NVFP4 (430 tensors) preserved; `token_embd`/`lm_head` BF16; scales/norms F32
- **Conversion:** `llama.cpp` `convert_hf_to_gguf.py --outtype bf16 --no-mtp` (main) and `--mmproj --outtype bf16` (vision)
- **File sizes:** main 19.60 GiB, mmproj 861 MiB

## License and attribution

Released under the **MIT** license, following the upstream models. This is an unofficial GGUF conversion
of `sakamakismile/Ornith-1.0-35B-NVFP4` (base model `deepreinforce-ai/Ornith-1.0-35B`; NVFP4 quantization by
`sakamakismile`). It is not affiliated with or endorsed by the original authors. Please consult the original
model cards for intended use and limitations.