Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,90 @@
+---
+language:
+  - en
+license: gemma
+tags:
+  - gemma4
+  - moe
+  - gguf
+  - reap
+  - pruned
+  - llama-cpp
+base_model: 0xSero/gemma-4-19b-a4b-it-REAP
+---
+
+# Gemma 4 19B-A4B-it REAP — GGUF Quantizations
+
+GGUF quantizations of [0xSero/gemma-4-19b-a4b-it-REAP](https://huggingface.co/0xSero/gemma-4-19b-a4b-it-REAP) — a **30% expert-pruned** Gemma 4 using [Cerebras REAP](https://github.com/cerebras/reap).
+
+## Available Quantizations
+
+| File                           | Quant  | Size       | BPW  | Use Case                                                 |
+| ------------------------------ | ------ | ---------- | ---- | -------------------------------------------------------- |
+| `gemma-4-19b-reap-Q4_K_M.gguf` | Q4_K_M | **12 GB**  | 5.32 | **Recommended.** Best quality/size tradeoff.             |
+| `gemma-4-19b-reap-Q3_K_S.gguf` | Q3_K_S | **8.4 GB** | 3.89 | Fits 12GB cards. Slight quality loss on technical terms. |
+
+## Performance (AMD RX 9070 XT, 16GB VRAM, Vulkan)
+
+| Metric                 | Original 26B Q4_K_M | REAP 19B Q4_K_M                                 |
+| ---------------------- | ------------------- | ----------------------------------------------- |
+| **Speed**              | 17 tok/s            | **130 tok/s**                                   |
+| **VRAM @ 2k ctx**      | 99%                 | **76%**                                         |
+| **Max context @ 16GB** | ~4k                 | **65k+**                                        |
+| **Quality**            | Baseline            | Indistinguishable on coding/reasoning/synthesis |
+
+The 7x speedup comes from crossing the VRAM comfort threshold — at 76% usage the GPU runs without memory pressure.
+
+## Quick Start
+
+### llama.cpp
+
+```bash
+# Download
+hf download vsark/gemma-4-19b-a4b-it-REAP-GGUF gemma-4-19b-reap-Q4_K_M.gguf
+
+# Run (IMPORTANT: use --reasoning off for Gemma 4)
+llama-server \
+    --model gemma-4-19b-reap-Q4_K_M.gguf \
+    --n-gpu-layers 99 \
+    --ctx-size 16384 \
+    --reasoning off \
+    --host 127.0.0.1 --port 8012
+```
+
+### Ollama
+
+```bash
+# Create a Modelfile
+echo 'FROM ./gemma-4-19b-reap-Q4_K_M.gguf
+PARAMETER num_ctx 16384' > Modelfile
+
+ollama create gemma4-19b-reap -f Modelfile
+ollama run gemma4-19b-reap
+```
+
+## Important Notes
+
+- **Use `--reasoning off`** with llama-server, or the model tries to emit thinking tokens
+- **Requires recent llama.cpp** with `gemma4` architecture support (older builds fail)
+- **Speculative decoding hurts** at these speeds (31 tok/s vs 130 without) — don't use it
+- Q3_K_S introduces occasional spelling errors on technical terms (e.g., "Affinity" → "Affity"). Use Q4_K_M for production.
+
+## About REAP
+
+REAP removes 30% of MoE experts (38 of 128 per layer) while keeping the same 8 active experts per token. Active parameter count is unchanged (~4B/token). The pruned experts were the least-used ones based on router gate values and activation norms across 22,000 calibration samples.
+
+See the [original model card](https://huggingface.co/0xSero/gemma-4-19b-a4b-it-REAP) for full details and benchmarks.
+
+## Conversion
+
+```bash
+python3 convert_hf_to_gguf.py ./gemma-4-19b-reap-bf16/ --outfile gemma-4-19b-reap-F16.gguf --outtype f16
+llama-quantize gemma-4-19b-reap-F16.gguf gemma-4-19b-reap-Q4_K_M.gguf Q4_K_M
+llama-quantize gemma-4-19b-reap-F16.gguf gemma-4-19b-reap-Q3_K_S.gguf Q3_K_S
+```
+
+## Credits
+
+- **REAP pruning**: [0xSero](https://huggingface.co/0xSero)
+- **Base model**: [Google Gemma 4](https://huggingface.co/google/gemma-4-26b-a4b-it)
+- **GGUF conversion + benchmarks**: [vsark](https://huggingface.co/vsark)