File size: 3,946 Bytes
2344020 609541b 2344020 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
language:
- en
license: gemma
tags:
- gemma4
- moe
- gguf
- reap
- pruned
- llama-cpp
base_model: 0xSero/gemma-4-19b-a4b-it-REAP
---
# Gemma 4 19B-A4B-it REAP — GGUF Quantizations
GGUF quantizations of [0xSero/gemma-4-19b-a4b-it-REAP](https://huggingface.co/0xSero/gemma-4-19b-a4b-it-REAP) — a **30% expert-pruned** Gemma 4 using [Cerebras REAP](https://github.com/cerebras/reap).
## Available Quantizations
| File | Quant | Size | BPW | Use Case |
| ------------------------------ | ------ | ---------- | ---- | -------------------------------------------------------- |
| `gemma-4-19b-reap-Q4_K_M.gguf` | Q4_K_M | **12 GB** | 5.32 | **Recommended.** Best quality/size tradeoff. |
| `gemma-4-19b-reap-Q3_K_S.gguf` | Q3_K_S | **8.4 GB** | 3.89 | Fits 12GB cards. Slight quality loss on technical terms. |
## Performance (AMD RX 9070 XT, 16GB VRAM, Vulkan)
| Metric | Original 26B Q4_K_M | REAP 19B Q4_K_M |
| ---------------------- | ------------------- | ----------------------------------------------- |
| **Speed** | 17 tok/s | **130 tok/s** |
| **VRAM @ 2k ctx** | 99% | **76%** |
| **Max context @ 16GB** | ~4k | **65k+** |
| **Quality** | Baseline | Indistinguishable on coding/reasoning/synthesis |
The 7x speedup comes from crossing the VRAM comfort threshold — at 76% usage the GPU runs without memory pressure.
## Quick Start
### llama.cpp
```bash
# Download
hf download vsark/gemma-4-19b-a4b-it-REAP-GGUF gemma-4-19b-reap-Q4_K_M.gguf
# Run (IMPORTANT: use --reasoning off for Gemma 4)
llama-server \
--model gemma-4-19b-reap-Q4_K_M.gguf \
--n-gpu-layers 99 \
--ctx-size 16384 \
--reasoning off \
--host 127.0.0.1 --port 8012
```
### Ollama
```bash
# Create a Modelfile
echo 'FROM ./gemma-4-19b-reap-Q4_K_M.gguf
PARAMETER num_ctx 16384' > Modelfile
ollama create gemma4-19b-reap -f Modelfile
ollama run gemma4-19b-reap
```
## Important Notes
- **Use `--reasoning off`** with llama-server, or the model tries to emit thinking tokens
- **Requires recent llama.cpp** with `gemma4` architecture support (older builds fail)
- **Speculative decoding hurts** at these speeds (31 tok/s vs 130 without) — don't use it
- Q3_K_S introduces occasional spelling errors on technical terms (e.g., "Affinity" → "Affity"). Use Q4_K_M for production.
## About REAP
REAP removes 30% of MoE experts (38 of 128 per layer) while keeping the same 8 active experts per token. Active parameter count is unchanged (~4B/token). The pruned experts were the least-used ones based on router gate values and activation norms across 22,000 calibration samples.
See the [original model card](https://huggingface.co/0xSero/gemma-4-19b-a4b-it-REAP) for full details and benchmarks.
## Conversion
```bash
python3 convert_hf_to_gguf.py ./gemma-4-19b-reap-bf16/ --outfile gemma-4-19b-reap-F16.gguf --outtype f16
llama-quantize gemma-4-19b-reap-F16.gguf gemma-4-19b-reap-Q4_K_M.gguf Q4_K_M
llama-quantize gemma-4-19b-reap-F16.gguf gemma-4-19b-reap-Q3_K_S.gguf Q3_K_S
```
## Verification
SHA256 checksums — if you reproduce the conversion from 0xSero's BF16 weights using the same llama.cpp version, you should get identical hashes:
```
77c579174b559a3d1812a8d8c03fa3a3ba514acc1ba54cd634ceaa783375d156 gemma-4-19b-reap-Q4_K_M.gguf
ed8e9acefa96cb97c00aaf0a79f2fe95893713148cf28d5cc8f3dad1d71dda6f gemma-4-19b-reap-Q3_K_S.gguf
```
## Credits
- **REAP pruning**: [0xSero](https://huggingface.co/0xSero)
- **Base model**: [Google Gemma 4](https://huggingface.co/google/gemma-4-26b-a4b-it)
- **GGUF conversion + benchmarks**: [vsark](https://huggingface.co/vsark)
|