Duplicate from empero-ai/Qwythos-9B-v2-GGUF

Co-authored-by: Empero <empero-ai@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,46 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+mmproj-Qwythos-9B-v2-BF16.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-MTP-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-MTP-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-MTP-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-MTP-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-MTP-BF16.gguf filter=lfs diff=lfs merge=lfs -text
+Qwythos-9B-v2-BF16.gguf filter=lfs diff=lfs merge=lfs -text

Qwythos-9B-v2-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:663e4694583caf7b3e5bdeb76c27aebb7ea86bcc2760419fea82e51e2d8087c7
+size 17920697088

Qwythos-9B-v2-MTP-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c46b135e2fef3c4d4b2a0e98394ae5ac5cdec54b1a5451c74ea568fbafb6075
+size 18407321280

Qwythos-9B-v2-MTP-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfdd00ac1c1dc9ced33f23817fb4282f2594067e02e34d82e3e63bc0ea275b05
+size 5903822528

Qwythos-9B-v2-MTP-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8ea0d2289793401a6ec546e713c885fba9e27d14cd70a36c570f975e383e659
+size 6710963904

Qwythos-9B-v2-MTP-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24271fb6e16b2b8f581d192cf77d947453d7852f965ce6801fefa6964bbea060
+size 7666069184

Qwythos-9B-v2-MTP-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d72e896721f2be249b2584fb06a007a8f8ce506e592ea54c103952303a5f11f
+size 9786060480

Qwythos-9B-v2-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0a588704f422b713eca29b2c1f192ae6f69aea3f9e7cb64f9ecdb76ff7a85f4
+size 5736063744

Qwythos-9B-v2-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc5251e8e3e87d58946522833eb896c84375a715b6f89c54cdd10082ecc6ea8d
+size 6523806464

Qwythos-9B-v2-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd39e148823f0bab858e946d603e4bf424240aae34435580be820f09c75ca379
+size 7458300672

Qwythos-9B-v2-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3324ac1a7260fbb484d10373df7d54d156ff6c4f5c577d87dbd0218928beab89
+size 9527501568

README.md

@@ -0,0 +1,199 @@
+---
+license: apache-2.0
+base_model: empero-ai/Qwythos-9B-v2
+base_model_relation: quantized
+language:
+- en
+pipeline_tag: image-text-to-text
+library_name: gguf
+tags:
+- gguf
+- llama.cpp
+- quantized
+- qwythos
+- qwen3.5
+- ftpo
+- reasoning
+- uncensored
+- long-context
+- 1M-context
+- function-calling
+- multimodal
+- vision
+---
+
+<p align="center">
+  <img src="https://huggingface.co/empero-ai/Qwythos-9B-v2/resolve/main/qwythos_v2.png" alt="Qwythos-9B-v2" width="640"/>
+</p>
+
+<p align="center"><b>Empero AI</b></p>
+
+# Qwythos-9B-v2-GGUF 
+
+GGUF quantizations of **[empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)** for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.
+
+**Qwythos-9B-v2 is the new and improved Qwythos** — all the deep chain-of-thought reasoning of the base Qwythos, with the **looping behavior fixed**. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (**6.7% → 0%**), the native **MTP head is restored**, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level.
+
+The fix uses **FTPO (Final-Token Preference Optimization)**: the exact token that *starts* a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched.
+
+For full training details, evaluation numbers, and sample generations, see the **[base model card](https://huggingface.co/empero-ai/Qwythos-9B-v2)**.
+
+---
+
+## What's new vs. the base Qwythos
+
+- 🔁 **Looping behavior eliminated** — repetition under greedy / low-temp decoding dropped **6.7% → 0%**. **Greedy decoding is now safe** — you no longer need `repeat-penalty` as a band-aid.
+- 🧩 **MTP head restored** — the native multi-token-prediction module is back in the `-MTP-` files, so speculative-decoding (`--spec-type draft-mtp`) works.
+- 🧠 **Reasoning preserved** — MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card).
+- 🪪 **Cleaner identity** — states who it is once, only when asked.
+- 🔓 Still intentionally **uncensored**, still **1M-token context** (YaRN), still **multimodal-capable** (Qwen3.5 vision tower).
+
+---
+
+## Files
+
+### Normal text weights — trunk only (32 blocks)
+
+| File | Quant | Size | Notes |
+|---|---|---|---|
+| `Qwythos-9B-v2-Q4_K_M.gguf` | Q4_K_M | 5.34 GiB / 5.74 GB | **recommended default** — smallest practical, good quality |
+| `Qwythos-9B-v2-Q5_K_M.gguf` | Q5_K_M | 6.08 GiB / 6.52 GB | balanced quality / size |
+| `Qwythos-9B-v2-Q6_K.gguf` | Q6_K | 6.95 GiB / 7.46 GB | high quality |
+| `Qwythos-9B-v2-Q8_0.gguf` | Q8_0 | 8.87 GiB / 9.53 GB | near-lossless |
+| `Qwythos-9B-v2-BF16.gguf` | BF16 | 16.69 GiB / 17.92 GB | full precision (conversion base) |
+
+If you don't know which to pick, **Q4_K_M is the right starting point.**
+
+### MTP-enabled text weights (33 blocks, `nextn_predict_layers = 1`)
+
+These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (`--spec-type draft-mtp`). The MTP matrices are retained at **Q8_0** in every quantized variant.
+
+| File | Quant | Size | Notes |
+|---|---|---|---|
+| `Qwythos-9B-v2-MTP-Q4_K_M.gguf` | Q4_K_M + MTP | 5.50 GiB / 5.90 GB | **recommended MTP default** |
+| `Qwythos-9B-v2-MTP-Q5_K_M.gguf` | Q5_K_M + MTP | 6.25 GiB / 6.71 GB | balanced quality / size |
+| `Qwythos-9B-v2-MTP-Q6_K.gguf` | Q6_K + MTP | 7.14 GiB / 7.67 GB | high quality |
+| `Qwythos-9B-v2-MTP-Q8_0.gguf` | Q8_0 + MTP | 9.11 GiB / 9.79 GB | near-lossless |
+| `Qwythos-9B-v2-MTP-BF16.gguf` | BF16 + MTP | 17.14 GiB / 18.41 GB | full precision (conversion base) |
+
+### Vision projector — for image input
+
+| File | Size | Notes |
+|---|---|---|
+| `mmproj-Qwythos-9B-v2-BF16.gguf` | 0.86 GiB / 0.92 GB | CLIP-style vision encoder + projector at **BF16** native precision; **required for images**, pairs with any text quant above |
+
+The vision tower is inherited **unchanged from Qwen3.5-9B** — it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B `mmproj-*.gguf`.
+
+---
+
+## Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)
+
+Qwythos is a **hybrid** model — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:
+
+| Quant | `ssm_alpha` | `ssm_beta` | `ssm_out` |
+|---|---|---|---|
+| **Q6_K** | Q8_0 | Q8_0 | Q8_0 |
+| **Q5_K_M** | Q8_0 | Q8_0 | Q6_K |
+| **Q4_K_M** | Q8_0 | Q8_0 | Q6_K |
+
+The remaining SSM state tensors (`ssm_a`, `ssm_conv1d`, `ssm_dt`, `ssm_norm`) are kept at **F32** by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant. `Q8_0` and `BF16` are uniform and need no overrides.
+
+---
+
+## Quick start
+
+### llama.cpp
+
+```bash
+llama-cli \
+  -m Qwythos-9B-v2-Q4_K_M.gguf \
+  -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \
+  -n 8192 \
+  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
+  -c 16384
+```
+
+Because v2's looping is trained out, `--repeat-penalty` is now optional and **greedy decoding (`--temp 0`) stays coherent.**
+
+### Ollama
+
+```bash
+ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
+```
+
+### LM Studio / jan / KoboldCpp
+
+Drop any `.gguf` into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.
+
+### MTP draft speculation
+
+```bash
+llama-server \
+  -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \
+  --spec-type draft-mtp \
+  --spec-draft-n-max 6 \
+  -c 16384 --port 8080
+```
+
+MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files.
+
+---
+
+## Vision (image input)
+
+Download a text quant **plus** the `mmproj-*.gguf`, then run llama.cpp's multimodal CLI/server:
+
+```bash
+llama-mtmd-cli \
+  -m Qwythos-9B-v2-Q4_K_M.gguf \
+  --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \
+  --image ./photo.jpg \
+  -p "Describe this image in detail." \
+  --temp 0.6 --top-p 0.95 --top-k 20 -c 16384
+```
+
+**Honest note:** all Qwythos training (base SFT and v2 FTPO) was **text-only** — the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.
+
+---
+
+## Sampling recommendations
+
+Qwythos is a reasoning model — every response opens with a `<think>...</think>` block before the answer.
+
+| Parameter | Value |
+|---|---|
+| `temperature` | 0.6 |
+| `top_p` | 0.95 |
+| `top_k` | 20 |
+| `repeat_penalty` | 1.05 (optional in v2) |
+| `max_new_tokens` | 16384 |
+
+Unlike the base Qwythos, **v2 does not loop under greedy / low-temperature decoding** — you can use `--temp 0` for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5's official thinking-mode recommendations for best quality.
+
+---
+
+## Long context (1M tokens)
+
+The GGUFs ship with YaRN rope-scaling baked in for a **1,048,576-token context window** (4× the 262,144 native). Set `-c` up to `1048576`; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256k–512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.
+
+---
+
+## Conversion & verification
+
+- Converted and quantized with **llama.cpp** (`convert_hf_to_gguf.py`, `llama-quantize`), architecture `qwen35`, GGUF v3.
+- MTP variants: default conversion (33-block, `nextn_predict_layers = 1`, 15 MTP tensors, MTP matrices pinned Q8_0). Normal variants: `--no-mtp` (32-block trunk-only). mmproj: `--mmproj --outtype bf16`.
+- Hybrid-precision overrides applied per the table above.
+- Structurally verified (arch / block count / `nextn` key / per-tensor types) and smoke-tested for load + coherent generation.
+- `shasum -a 256 -c SHA256SUMS` covers all 11 artifacts.
+
+---
+
+## License & acknowledgements
+
+Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.
+
+- Developed and released by **Empero AI**
+- Base model: **Qwen3.5-9B** (Alibaba Qwen team)
+- Looping fixed with **FTPO (Final-Token Preference Optimization)**
+- Quantization: **llama.cpp** (ggml-org)
+- HF model: [empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)

SHA256SUMS

@@ -0,0 +1,11 @@
+663e4694583caf7b3e5bdeb76c27aebb7ea86bcc2760419fea82e51e2d8087c7  Qwythos-9B-v2-BF16.gguf
+2c46b135e2fef3c4d4b2a0e98394ae5ac5cdec54b1a5451c74ea568fbafb6075  Qwythos-9B-v2-MTP-BF16.gguf
+cfdd00ac1c1dc9ced33f23817fb4282f2594067e02e34d82e3e63bc0ea275b05  Qwythos-9B-v2-MTP-Q4_K_M.gguf
+d8ea0d2289793401a6ec546e713c885fba9e27d14cd70a36c570f975e383e659  Qwythos-9B-v2-MTP-Q5_K_M.gguf
+24271fb6e16b2b8f581d192cf77d947453d7852f965ce6801fefa6964bbea060  Qwythos-9B-v2-MTP-Q6_K.gguf
+6d72e896721f2be249b2584fb06a007a8f8ce506e592ea54c103952303a5f11f  Qwythos-9B-v2-MTP-Q8_0.gguf
+c0a588704f422b713eca29b2c1f192ae6f69aea3f9e7cb64f9ecdb76ff7a85f4  Qwythos-9B-v2-Q4_K_M.gguf
+fc5251e8e3e87d58946522833eb896c84375a715b6f89c54cdd10082ecc6ea8d  Qwythos-9B-v2-Q5_K_M.gguf
+dd39e148823f0bab858e946d603e4bf424240aae34435580be820f09c75ca379  Qwythos-9B-v2-Q6_K.gguf
+3324ac1a7260fbb484d10373df7d54d156ff6c4f5c577d87dbd0218928beab89  Qwythos-9B-v2-Q8_0.gguf
+0d1687cb33124c78acab788b342d4a2eaf85b3035e87c3abe4ee9d0b84ddb4f5  mmproj-Qwythos-9B-v2-BF16.gguf

mmproj-Qwythos-9B-v2-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d1687cb33124c78acab788b342d4a2eaf85b3035e87c3abe4ee9d0b84ddb4f5
+size 921704512