BillFan666/Ornith-1.5-35B-A3B-ADQ4-Shisa12K-MTP-GGUF

Ornith 1.5 35B A3B AD-Q4 + Shisa 12K MTP GGUF

Single-file language-model GGUF combining AtomicChat's model-specific AD-Q4_K-IQ4_XS Ornith target with Shisa's 12K KL-distilled MTP head, quantized to Q4_0. It was assembled and tuned for llama.cpp on an NVIDIA RTX 5060 Ti 16 GB while retaining the model's full 262,144-token context allocation. The file itself does not contain a vision projector; compatibility with the official external Ornith BF16 mmproj is documented and verified below.

中文摘要：这是 AtomicChat AD-Q4 主模型与 Shisa 12K 蒸馏 MTP 头的单文件 GGUF 合并版。README 下面完整记录了权重来源、版本、量化/合并方法、运行补丁、 262K 配置，以及 RTX 5060 Ti 16GB 的实测速度和显存边界。

This is a community-built derivative, not an official release from Ornith, AtomicChat, or Shisa. Use a recent llama.cpp build with qwen35moe MTP speculative-decoding support.

File

File	Size	Contents
Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS-Shisa-12K-MTP-Q4_0.gguf	20,602,882,240 bytes (19.19 GiB)	40-layer target + one appended NextN/MTP layer

• Architecture: qwen35moe
• Target tensor descriptors: 733
• Appended MTP descriptors: 20
• Total tensor descriptors: 753
• GGUF block count after graft: 41
• nextn_predict_layers: 1
• SHA-256: 297c3f150723426fb983dbbe891dcdd97b47e4e40db48fd2ba0564d31ad29676

Exact model sources and provenance

Target model

• Repository: AtomicChat/Ornith-1.5-35B-A3B-GGUF
• Revision: 7aa8fc1d9b861d797880f4a341166d4bb3439f74
• File: Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS.gguf
• Declared license: MIT
• Upstream base: ornith-ai/Ornith-1.5-35B-A3B
• Metadata revision used during conversion: e4dfb35a93d4b6822a811a7676f3488514abe7e2

The target's AtomicChat mixed AD-Q4_K/IQ4_XS tensors were copied byte for byte. They were not requantized during the graft.

MTP draft head

• Repository: shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY
• Revision: 2b19b31bfe1659c6b0d9459ec3cbd87e34a322ef
• File: model-mtp.safetensors
• Source payload: 844,640,768 parameters / 1.689 GB, BF16
• Training: Shisa's final 12K, two-epoch, code-heavy KL-distilled MTP head
• Declared license: Apache-2.0; see LICENSE and NOTICE
• Head lineage: initialized from the compatible MTP head in Qwen/Qwen3.6-35B-A3B, then KL-distilled against Ornith-1.5 hidden states by Shisa.

The Shisa head was converted to GGUF BF16, quantized separately to Q4_0, and then appended to the AtomicChat target. Descriptor counts and representative tensor payload hashes were checked after grafting.

llama.cpp compatibility

The tested build was based on llama.cpp commit 4df29be4f4c3673f428170fda944a5b19f743bb8:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j --target llama-server llama-cli

The local source tree also contained an experimental 27-insertion/1-deletion d2t compatibility change for other trimmed-vocabulary draft models. An exhaustive tensor inspection confirms that this GGUF contains no d2t tensor, so that conditional code path is never executed and no local source patch is required for this artifact.

Tested full-context serving configuration

The following is the exact performance-oriented shape used for the results below. No fixed --n-cpu-moe or --gpu-layers value is used: --fit reserves both KV caches and compute buffers first, then assigns the safe expert subset.

MODEL=Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS-Shisa-12K-MTP-Q4_0.gguf

./build/bin/llama-server \
  --model "$MODEL" \
  --ctx-size 262144 \
  --parallel 1 \
  --n-predict 32768 \
  --batch-size 512 \
  --ubatch-size 128 \
  --split-mode none \
  --flash-attn on \
  --backend-sampling \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 1 \
  --spec-draft-n-min 0 \
  --draft-p-min 0 \
  --draft-p-split 0.10 \
  --cache-type-k-draft q8_0 \
  --cache-type-v-draft q8_0 \
  --threads 8 \
  --threads-batch 16 \
  --load-mode none \
  --fit on \
  --fit-target 1024

Why MTP1: on this machine MTP1 was faster than longer speculative depths. Setting p-min=0 also beat the tested 0.30 and 0.60 thresholds. Q8 target and draft KV were retained for long-context quality.

Optional vision projector (not included)

Vision was validated with the official external projector; it is deliberately not duplicated in this repository:

• Repository: ornith-ai/Ornith-1.5-35B-A3B-GGUF
• Revision: fbbaed45c2f0e200276ffa51701a24d45dc7f57e
• File: mmproj-Ornith-1.5-35B-BF16.gguf
• Size: 902,822,016 bytes
• SHA-256: d9ce31026d1cb1f3f8d5152e2e2a014d9d2b302b6c93a7dc07bb0a0487f52837
• Declared license: MIT

On a 16GB card, keep the projector in host RAM to preserve VRAM for Q8 KV and expert weights:

  --mmproj mmproj-Ornith-1.5-35B-BF16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024

The remaining model, MTP, KV, fit, batch, and thread settings were identical to the 262K command above.

RTX 5060 Ti 16GB results

Test system

• GPU: NVIDIA GeForce RTX 5060 Ti 16 GB, CUDA compute capability sm_120
• CPU: Intel Core i7-10700KF, 8 cores / 16 threads
• System RAM: 46 GiB available to Linux
• Runtime: llama.cpp-based CUDA build at 4df29be4f; an unrelated conditional d2t source change was present but inactive for this GGUF
• Concurrency: one sequence
• Workload: text benchmarks plus the separate CPU-resident projector check below

Memory placement at 262K

• llama.cpp reported 42/42 logical allocation units offloaded to CUDA, but this does not mean every expert tensor was in VRAM.
• With fit-target=1024, 23 units contained expert-weight overflow to host RAM.
• Approximately 19 allocation units were completely GPU resident. The first boundary target block was partial; later target/MTP expert matrices stayed in host RAM while attention, linear-attention/SSM, norms, routers, embeddings, and output tensors stayed on the GPU.
• Target Q8 KV allocation: 2,720 MiB.
• MTP Q8 KV allocation: 272 MiB.
• Recurrent-state allocation: 125.62 MiB.
• Idle/short-prompt VRAM: approximately 14,680 MiB.
• Observed VRAM during a 27,919-token prefill: approximately 14,858 MiB.

Throughput

All numbers are local single-user measurements and should not be treated as universal hardware claims.

Mode	Workload	Decode throughput
Plain autoregressive	four fixed 256-token outputs	~50.5 tok/s average
MTP1, p-min=0	four fixed 256-token outputs	~63.7 tok/s average
MTP2, p-min=0	four fixed 256-token outputs	~63.6 tok/s average
MTP3, p-min=0	four fixed 256-token outputs	~59.0 tok/s average
MTP1, p-min=0	four fixed 512-token outputs	~62.8 tok/s average
MTP1 after final Q8/Q8 reload	four additional fixed 256-token outputs	63.47 tok/s average

The selected MTP1 profile improved the original four-prompt average by about 26% over plain AR. Acceptance is prompt-dependent; the long-prompt test below accepted 112 of 142 draft tokens (78.9%).

Longer prompt check

• Prompt: 27,919 tokens
• Prompt ingestion: 136.5 tok/s
• Generated: 256 tokens
• Decode: 55.15 tok/s
• Draft acceptance: 112 / 142 (78.9%)
• Result: completed without OOM or server restart

This is a 27.9K-token performance check inside a fully allocated 262K context, not a claim that decode was benchmarked with 262K tokens already populated.

Vision compatibility check

A 326KB JPEG of the Statue of Liberty was sent as an embedded base64 image, so the server did not receive a descriptive remote URL. With thinking disabled, the model returned:

The main landmark is the Statue of Liberty, located in New York City.

• Image + text prompt: 1,687 tokens
• CPU-projector/prompt processing: 33.43 seconds / 50.46 tok/s
• Final answer: 16 tokens / 56.75 tok/s
• MTP acceptance on the short final answer: 8 / 8
• A separate 128-token thinking run decoded at 60.61 tok/s and accepted 57 / 69 MTP drafts (82.6%)
• Peak observed GPU allocation: approximately 14,862 MiB
• Result: correct landmark/city identification; no OOM or server restart

16GB stability boundary

An aggressive all-Q8 experiment reduced fit-target to zero and moved roughly three more complete expert units to the GPU. It loaded at about 15,650 MiB and rose to about 15,790 MiB during generation, then failed with CUDA OOM while instantiating a CUDA graph on the third short request. Do not copy that setting on a display-attached 16GB card. The published 1,024 MiB fit margin is the stable profile.

Validation performed

• Structural check: 733 target + 20 appended MTP descriptors = 753 tensors.
• Target descriptors and weights were preserved through grafting.
• Representative target and MTP payload hashes matched their source GGUFs.
• CPU-only llama.cpp load/inference passed.
• CUDA load, speculative-context initialization, and generation passed.
• Native 262,144-token context allocation passed with Q8 target and draft KV.

Limitations

• No mmproj is included. Vision requires the separately downloaded official projector cited above.
• Vision validation currently covers one landmark image, not a broad OCR, grounding, chart, document, or multimodal safety benchmark.
• The MTP head was trained by Shisa with a 2,048-token training window. Shisa measured longer-context acceptance, but the training distribution does not directly cover 256K inputs.
• The target is quantized; see AtomicChat's model card for KL and top-1 measurements against BF16.
• Speculative acceptance and throughput depend strongly on prompt distribution, sampling parameters, runtime revision, and hardware.
• Use a llama.cpp revision with Qwen3.5 MoE draft-mtp support. This GGUF does not require the separate trimmed-vocabulary d2t compatibility patch.
• This artifact has not been independently evaluated on downstream capability, safety, or multimodal benchmarks.

Licenses and attribution

This artifact combines components with different declared licenses:

• AtomicChat target GGUF and upstream Ornith metadata: MIT as declared by their Hugging Face repositories.
• Shisa MTP head and its Qwen MTP lineage: Apache License 2.0. The Shisa LICENSE and NOTICE are redistributed in this repository.

Users are responsible for reviewing and complying with all upstream licenses, model cards, acceptable-use requirements, and applicable law. Attribution in this README is informational and does not replace the upstream terms.

Acknowledgements

All model research, target quantization, and MTP distillation credit belongs to the respective Ornith, AtomicChat, Shisa, and Qwen authors. This repository only packages the cited target and MTP head into a tested llama.cpp GGUF combination.