Instructions to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Use Docker

docker model run hf.co/BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Ollama
How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with Ollama:
```
ollama run hf.co/BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
```
Unsloth Desktop

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0
```

Lemonade

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Run and chat with the model

lemonade run user.Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF / README.md

BillFan666

Add model card with sources, configuration, and RTX 5060 Ti benchmarks

72438f0 verified 6 days ago

9.88 kB

license: other
license_name: mit-target-plus-apache-2.0-mtp
base_model:
  - ornith-ai/Ornith-1.5-35B-A3B
  - AtomicChat/Ornith-1.5-35B-A3B-GGUF
  - shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY
library_name: llama.cpp
pipeline_tag: text-generation
tags:
  - gguf
  - llama.cpp
  - qwen35moe
  - mixture-of-experts
  - speculative-decoding
  - mtp
  - quantized
  - blackwell
language:
  - en
  - zh
  - ja
  - ko

Ornith 1.5 35B A3B AD-Q4 + Shisa 12K MTP GGUF

Single-file, text-only GGUF combining AtomicChat's model-specific AD-Q4_K-IQ4_XS Ornith target with Shisa's 12K KL-distilled MTP head, quantized to Q4_0. It was assembled and tuned for llama.cpp on an NVIDIA RTX 5060 Ti 16 GB while retaining the model's full 262,144-token context allocation.

中文摘要：这是 AtomicChat AD-Q4 主模型与 Shisa 12K 蒸馏 MTP 头的单文件 GGUF 合并版。README 下面完整记录了权重来源、版本、量化/合并方法、运行补丁、 262K 配置，以及 RTX 5060 Ti 16GB 的实测速度和显存边界。

This is a community-built derivative, not an official release from Ornith, AtomicChat, or Shisa. The tested llama.cpp revision needs the included llama.cpp-qwen35-d2t-mtp.patch for the MTP head's trimmed draft vocabulary.

File

File	Size	Contents
`Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS-Shisa-12K-MTP-Q4_0.gguf`	20,602,882,240 bytes (19.19 GiB)	40-layer target + one appended NextN/MTP layer

Architecture: qwen35moe
Target tensor descriptors: 733
Appended MTP descriptors: 20
Total tensor descriptors: 753
GGUF block count after graft: 41
nextn_predict_layers: 1
SHA-256: 297c3f150723426fb983dbbe891dcdd97b47e4e40db48fd2ba0564d31ad29676

Exact model sources and provenance

Target model

Repository: AtomicChat/Ornith-1.5-35B-A3B-GGUF
Revision: 7aa8fc1d9b861d797880f4a341166d4bb3439f74
File: Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS.gguf
Declared license: MIT
Upstream base: ornith-ai/Ornith-1.5-35B-A3B
Metadata revision used during conversion: e4dfb35a93d4b6822a811a7676f3488514abe7e2

The target's AtomicChat mixed AD-Q4_K/IQ4_XS tensors were copied byte for byte. They were not requantized during the graft.

MTP draft head

Repository: shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY
Revision: 2b19b31bfe1659c6b0d9459ec3cbd87e34a322ef
File: model-mtp.safetensors
Source payload: 844,640,768 parameters / 1.689 GB, BF16
Training: Shisa's final 12K, two-epoch, code-heavy KL-distilled MTP head
Declared license: Apache-2.0; see LICENSE and NOTICE
Head lineage: initialized from the compatible MTP head in Qwen/Qwen3.6-35B-A3B, then KL-distilled against Ornith-1.5 hidden states by Shisa.

The Shisa head was converted to GGUF BF16, quantized separately to Q4_0, and then appended to the AtomicChat target. Descriptor counts and representative tensor payload hashes were checked after grafting.

llama.cpp compatibility patch

The Shisa head uses a trimmed draft vocabulary plus a d2t mapping back to the full 248,320-token target vocabulary. The tested build was based on llama.cpp commit 4df29be4f4c3673f428170fda944a5b19f743bb8 with the included patch:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
git apply ../llama.cpp-qwen35-d2t-mtp.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j --target llama-server llama-cli

Later llama.cpp revisions may already contain equivalent support. Confirm that your build recognizes d2t, initializes the speculative context, and returns valid full-vocabulary logits before relying on it.

Tested full-context serving configuration

The following is the exact performance-oriented shape used for the results below. No fixed --n-cpu-moe or --gpu-layers value is used: --fit reserves both KV caches and compute buffers first, then assigns the safe expert subset.

MODEL=Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS-Shisa-12K-MTP-Q4_0.gguf

./build/bin/llama-server \
  --model "$MODEL" \
  --ctx-size 262144 \
  --parallel 1 \
  --n-predict 32768 \
  --batch-size 512 \
  --ubatch-size 128 \
  --split-mode none \
  --flash-attn on \
  --backend-sampling \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 1 \
  --spec-draft-n-min 0 \
  --draft-p-min 0 \
  --draft-p-split 0.10 \
  --cache-type-k-draft q8_0 \
  --cache-type-v-draft q8_0 \
  --threads 8 \
  --threads-batch 16 \
  --load-mode none \
  --fit on \
  --fit-target 1024

Why MTP1: on this machine MTP1 was faster than longer speculative depths. Setting p-min=0 also beat the tested 0.30 and 0.60 thresholds. Q8 target and draft KV were retained for long-context quality.

RTX 5060 Ti 16GB results

Test system

GPU: NVIDIA GeForce RTX 5060 Ti 16 GB, CUDA compute capability sm_120
CPU: Intel Core i7-10700KF, 8 cores / 16 threads
System RAM: 46 GiB available to Linux
Runtime: locally patched llama.cpp 4df29be4f, CUDA build
Concurrency: one sequence
Workload: text only; no multimodal projector

Memory placement at 262K

llama.cpp reported 42/42 logical allocation units offloaded to CUDA, but this does not mean every expert tensor was in VRAM.
With fit-target=1024, 23 units contained expert-weight overflow to host RAM.
Approximately 19 allocation units were completely GPU resident. The first boundary target block was partial; later target/MTP expert matrices stayed in host RAM while attention, linear-attention/SSM, norms, routers, embeddings, and output tensors stayed on the GPU.
Target Q8 KV allocation: 2,720 MiB.
MTP Q8 KV allocation: 272 MiB.
Recurrent-state allocation: 125.62 MiB.
Idle/short-prompt VRAM: approximately 14,680 MiB.
Observed VRAM during a 27,919-token prefill: approximately 14,858 MiB.

Throughput

All numbers are local single-user measurements and should not be treated as universal hardware claims.

Mode	Workload	Decode throughput
Plain autoregressive	four fixed 256-token outputs	~50.5 tok/s average
MTP1, `p-min=0`	four fixed 256-token outputs	~63.7 tok/s average
MTP2, `p-min=0`	four fixed 256-token outputs	~63.6 tok/s average
MTP3, `p-min=0`	four fixed 256-token outputs	~59.0 tok/s average
MTP1, `p-min=0`	four fixed 512-token outputs	~62.8 tok/s average
MTP1 after final Q8/Q8 reload	four additional fixed 256-token outputs	63.47 tok/s average

The selected MTP1 profile improved the original four-prompt average by about 26% over plain AR. Acceptance is prompt-dependent; the long-prompt test below accepted 112 of 142 draft tokens (78.9%).

Longer prompt check

Prompt: 27,919 tokens
Prompt ingestion: 136.5 tok/s
Generated: 256 tokens
Decode: 55.15 tok/s
Draft acceptance: 112 / 142 (78.9%)
Result: completed without OOM or server restart

This is a 27.9K-token performance check inside a fully allocated 262K context, not a claim that decode was benchmarked with 262K tokens already populated.

16GB stability boundary

An aggressive all-Q8 experiment reduced fit-target to zero and moved roughly three more complete expert units to the GPU. It loaded at about 15,650 MiB and rose to about 15,790 MiB during generation, then failed with CUDA OOM while instantiating a CUDA graph on the third short request. Do not copy that setting on a display-attached 16GB card. The published 1,024 MiB fit margin is the stable profile.

Validation performed

Structural check: 733 target + 20 appended MTP descriptors = 753 tensors.
Target descriptors and weights were preserved through grafting.
Representative target and MTP payload hashes matched their source GGUFs.
CPU-only llama.cpp load/inference passed.
CUDA load, speculative-context initialization, and generation passed.
Native 262,144-token context allocation passed with Q8 target and draft KV.

Limitations

Text-only testing; no mmproj is included.
The MTP head was trained by Shisa with a 2,048-token training window. Shisa measured longer-context acceptance, but the training distribution does not directly cover 256K inputs.
The target is quantized; see AtomicChat's model card for KL and top-1 measurements against BF16.
Speculative acceptance and throughput depend strongly on prompt distribution, sampling parameters, runtime revision, and hardware.
The included llama.cpp patch is required for the exact tested revision. A stock or older runtime may fail to load the draft head or may mishandle its vocabulary mapping.
This artifact has not been independently evaluated on downstream capability, safety, or multimodal benchmarks.

Licenses and attribution

This artifact combines components with different declared licenses:

AtomicChat target GGUF and upstream Ornith metadata: MIT as declared by their Hugging Face repositories.
Shisa MTP head and its Qwen MTP lineage: Apache License 2.0. The Shisa LICENSE and NOTICE are redistributed in this repository.

Users are responsible for reviewing and complying with all upstream licenses, model cards, acceptable-use requirements, and applicable law. Attribution in this README is informational and does not replace the upstream terms.

Acknowledgements

All model research, target quantization, and MTP distillation credit belongs to the respective Ornith, AtomicChat, Shisa, and Qwen authors. This repository only packages the cited target and MTP head into a tested llama.cpp GGUF combination.