Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-GGUF

GGUF conversion of AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4 for use with llama.cpp.

This is an NVFP4-quantized standalone Mixture-of-Experts language model.

This repository preserves NVIDIA's original NVFP4 tensors in GGUF format. It does not embed an MTP block. For native MTP speculative decoding, use a compatible MTP-only GGUF through --model-draft.

Model Details

• Source model: AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4
• Base model: deepreinforce-ai/Ornith-1.0-35B
• Format: GGUF
• Quantization: NVIDIA NVFP4
• Architecture: Qwen3.6 Mixture-of-Experts with hybrid attention
• Parameters: 35B total, approximately 3B activated per token
• Purpose: Local inference with llama.cpp

NVIDIA quantized the weights and activations of linear operators inside the MoE transformer blocks to NVFP4. Other tensors may remain in higher-precision formats.

This repository contains the complete target model. It is not an MTP-only, EAGLE3, or DFlash draft checkpoint.

Compatibility

A recent version of llama.cpp with Qwen3.6 MoE and NVFP4 support is required.

Tested with:

• Windows
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp CUDA backend
• External Qwen3.6 MTP-only draft GGUF via --model-draft

Older llama.cpp builds may fail to recognize the nvfp4 tensor type, may not correctly load associated scale tensors, or may lack compatible Qwen3.6 MoE support.

Performance may vary with llama.cpp build, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.

Usage

llama-server, standard decoding

llama-server \
  -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4.gguf \
  --flash-attn on \
  -ngl 99

llama-server with external MTP-only draft model

Recommended for native MTP speculative decoding:

llama-server \
  -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4.gguf \
  --model-draft Qwen3.6-35B-A3B-MTP-ONLY-Q6_K.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --flash-attn on \
  -ngl 99

llama-cli

llama-cli \
  -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4.gguf \
  --flash-attn on \
  -ngl 99

Multi-GPU example

llama-server \
  -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4.gguf \
  --model-draft Qwen3.6-35B-A3B-MTP-ONLY-Q6_K.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --split-mode layer \
  --tensor-split 1,1 \
  --flash-attn on \
  -ngl 99

The best tensor split depends on available VRAM, GPU speed, PCIe topology, context length, and KV-cache placement. An even split is only a starting point.

Speculation Comparison

Mode	Settings	Aggregate acceptance	Wall time	Notes
No MTP	Standard decoding	n/a	21.88 s	Baseline. Around 85–88 tok/s in this benchmark.
Native MTP / External MTP-only GGUF	--model-draft Qwen3.6-35B-A3B-MTP-ONLY-Q6_K.gguf, n_max=3	95.09%	22.33 s	Recommended. Best balance of acceptance and throughput in local tests.
Embedded MTP	Grafted blk.40.* into the main GGUF	93.17%	24.51 s	Functional, but slower than external --model-draft.
DFlash	n_max=3, p_min=0.65	88.63%	22.40 s	Aggressive setting. Similar wall time, lower acceptance.
DFlash	n_max=4, p_min=0.75	92.46%	22.84 s	Best DFlash balance in local tests.
DFlash	n_max=3, p_min=0.85	96.06%	25.25 s	Conservative setting. Higher acceptance, but slower.

In these tests, native MTP via an external MTP-only GGUF remained the best overall option for this Qwen3.6 MoE fine-tune.

DFlash became tunable after llama.cpp b9867 (spec: support spec-draft-p-min in DFlash). Increasing --spec-draft-p-min reduced draft volume and increased acceptance, but also reduced throughput.

For this model, the best tested DFlash setting was:

--spec-type draft-dflash
--spec-draft-n-max 4
--spec-draft-p-min 0.75

However, external MTP-only GGUF still provided the best balance:

--model-draft Qwen3.6-35B-A3B-MTP-ONLY-Q6_K.gguf
--spec-type draft-mtp
--spec-draft-n-max 3

Even though the non-speculative baseline shows a slightly shorter total wall time in this short benchmark, it only runs around 85–88 tok/s and does not benefit from speculative decoding. Per-request token throughput and acceptance rate are more useful than wall time alone when comparing speculation methods.

MTP Recommendation

For most users, a standalone MTP-only draft model is recommended instead of embedding the MTP tensors into this GGUF.

Recommended draft source:

https://huggingface.co/a4lg/Qwen3.6-35B-A3B-MTP-ONLY-GGUF

The MTP-only draft can be used with:

--model-draft
--spec-type draft-mtp

Based on my llama.cpp benchmarks, for Qwen3.5/Qwen3.6 MoE models it is currently preferable to keep the native MTP block as a separate --model-draft GGUF rather than embedding it directly into the main GGUF.

In my tests, the external MTP-only draft model consistently delivered higher throughput than the embedded MTP version, even when running on a single GPU.

This design also makes it easier to maintain and distribute MoE fine-tunes, since the MTP block can be updated independently from the main GGUF.

The same MTP-only GGUF can therefore be reused across multiple compatible Qwen3.5/Qwen3.6 MoE fine-tunes without rebuilding each GGUF individually.

Current recommendation:

• Standard GGUF + --model-draft → Recommended

Suggested MTP Settings

A reasonable general-purpose starting point is:

--spec-draft-n-max 3

For translation, role-play, creative writing, conversational output, or open-ended explanations, start with:

--spec-draft-n-max 2

For JSON, fixed templates, repeated patterns, and deterministic code completion, try:

--spec-draft-n-max 4

Conversion

Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:

python convert_hf_to_gguf.py \
  /path/to/Qwen3.6-35B-A3B-NVFP4 \
  --outfile Qwen3.6-35B-A3B-NVFP4.gguf \
  --outtype auto \
  --verbose

Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.

No neural-network weights were retrained. The original NVFP4 weights and their associated scale tensors were converted to GGUF-compatible representations.

File Size

The converted GGUF may be larger than the original Hugging Face ModelOpt checkpoint.

The conversion does not copy the original safetensors byte-for-byte. NVFP4 weights are repacked into the GGML layout, and the GGUF stores the scale and input-scale tensors required by llama.cpp.

A larger GGUF file does not mean that the model was converted to FP16 or to a conventional Q4 format. The tensor dump should still show the main linear weights as:

type = nvfp4

Windows reports file sizes using binary units even though File Explorer labels them as GB. For example, a file shown as 28.2 GB by Hugging Face may appear as approximately 26.2 GB in Windows.

File size should not be treated as the exact amount of VRAM required at runtime. Additional memory is needed for the KV cache, CUDA context, compute buffers, graph workspace, and speculative decoding.

Verifying the GGUF

The converted file can be checked with llama-gguf:

.\llama-gguf.exe E:\HF_MODELS\Qwen3.6-35B-A3B-NVFP4.gguf r

A successful conversion should show NVFP4 tensors and their associated scale tensors, for example:

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.

Notes

• The model is a 35B-total-parameter MoE with approximately 3B parameters activated per token; runtime memory requirements still depend on the full stored checkpoint rather than only the active parameter count.
• Model size, VRAM requirements, and speed should not be estimated as though every parameter were stored as a plain 4-bit scalar. NVFP4 uses block scales and leaves some tensors at higher precision.
• Native MTP accelerates token generation but does not improve prompt-prefill speed in the same way.
• Very short benchmark outputs are more sensitive to run-to-run variance.
• Results are specific to the tested hardware, llama.cpp build, prompts, runtime options, and context configuration.

Related Projects

• Embedded MTP experimental version: Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-MTP-GGUF
• Grafting utility: gguf-graft-mtp

Credits

• Qwen Team / Alibaba Cloud — Qwen3.6-35B-A3B
• ggml-org — llama.cpp, GGUF, NVFP4 inference support, and native MTP support

License

The source model is distributed under the Apache License 2.0.

Users should review the upstream AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4 and deepreinforce-ai/Ornith-1.0-35B model cards before redistribution or commercial use.