README.md

---

license: apache-2.0
base_model:
  - RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3
  - unsloth/gemma-4-26B-A4B-it-GGUF
base_model_relation: quantized
library_name: llama.cpp
tags:
  - gguf
  - llama.cpp
  - eagle3
  - speculative-decoding
  - speculator
  - draft-model
  - gemma-4
  - gemma
  - moe
  - redhatai
  - unsloth
pipeline_tag: text-generation

---


# Gemma 4 26B-A4B IT EAGLE3 Speculator GGUF

This repository contains GGUF conversions and quantizations of **RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3** for use with **llama.cpp EAGLE3 speculative decoding**.

> [!IMPORTANT]
> This is **not a standalone chat model**. It is an **EAGLE3 draft/speculator model** and must be used together with the matching target/verifier model.

* Target model: `unsloth/gemma-4-26B-A4B-it-GGUF`
* Speculator source: `RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3`
* Runtime: llama.cpp with `--spec-type draft-eagle3`

## Files

| File                                               | Type                             | Notes                                                         |
| -------------------------------------------------- | -------------------------------- | ------------------------------------------------------------- |
| `gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf`    | EAGLE3 speculator GGUF           | Converted from the original RedHatAI safetensors checkpoint   |
| `gemma-4-26B-A4B-it-speculator.eagle3-Q8_0.gguf`   | Quantized EAGLE3 speculator GGUF | Quantized from the F16 GGUF                                   |
| `gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf` | Quantized EAGLE3 speculator GGUF | Quantized from the F16 GGUF; may be faster for draft decoding |

## Usage with llama.cpp

Example:

```bash

llama-server \

  -m gemma-4-26B-A4B-it-Q4_K_M.gguf \

  -md gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf \

  --spec-type draft-eagle3 \

  --spec-draft-n-max 4 \

  --spec-draft-p-min 0.5 \

  -c 32768 \

  -ngl 99 \

  -fa on

```

Windows CMD example:

```cmd

llama-server.exe ^

  -m gemma-4-26B-A4B-it-Q4_K_M.gguf ^

  -md gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf ^

  --spec-type draft-eagle3 ^

  --spec-draft-n-max 4 ^

  --spec-draft-p-min 0.5 ^

  -c 32768 ^

  -ngl 99 ^

  -fa on

```

PowerShell example:

```powershell

.\llama-server.exe `

  -m "gemma-4-26B-A4B-it-Q4_K_M.gguf" `

  -md "gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf" `

  --spec-type draft-eagle3 `

  --spec-draft-n-max 4 `

  --spec-draft-p-min 0.5 `

  -c 32768 `

  -ngl 99 `

  -fa on

```

## Important Notes

This GGUF file is only the **draft/speculator model**. You still need a compatible GGUF of the target model, such as `unsloth/gemma-4-26B-A4B-it-GGUF`.

Do **not** use this speculator with unrelated models such as Gemma 4 12B, Gemma 4 31B, Gemma 3, Qwen, Llama, Mistral, or other non-matching models. EAGLE3 speculators are target-specific.

Even small differences in the target model, prompt format, quantization, or runtime settings may affect draft acceptance rate and overall speed.

## Tested Configuration

Tested with:

* Runtime: llama.cpp with EAGLE3 support
* Target model: `unsloth/gemma-4-26B-A4B-it-GGUF`
* Draft model: this EAGLE3 GGUF
* Example settings:

  * `--spec-type draft-eagle3`
  * `--spec-draft-n-max 4`
  * `--spec-draft-p-min 0.5`

Local benchmark observations may vary depending on GPU, quantization, context length, batch size, sampling settings, and prompt type.

## Benchmark Notes

In local testing, Gemma 4 26B-A4B IT without EAGLE3 already showed strong baseline decoding speed.

With EAGLE3 enabled, the draft acceptance rate was around `0.70` in local testing, with stronger gains on structured or predictable tasks such as:

* JSON output
* stepwise math
* code completion
* summarization
* long reasoning
* repeated pattern generation

It was less effective on some open-ended or language-sensitive tasks such as:

* translation
* creative writing
* general explanation
* some factual QA prompts

On this model, EAGLE3 may be useful for structured output, agent/tool-style responses, code completion, and predictable formats. For general chat, translation, roleplay, or creative writing, the non-speculative baseline may be competitive or more consistent.

On smaller VRAM setups, the extra draft/speculator model may reduce the practical benefit of EAGLE3. In those cases, native MTP models or the base Gemma 4 26B-A4B model without speculative decoding may be more efficient.

## Conversion

Converted with llama.cpp `convert_hf_to_gguf.py` using the original speculator repository and the matching target model directory.

Example conversion command:

```bash

python convert_hf_to_gguf.py \

  RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3 \

  --outtype f16 \

  --target-model-dir gemma-4-26B-A4B-it \

  --outfile gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf

```

PowerShell example:

```powershell

python .\convert_hf_to_gguf.py `

  "E:\OLLAMA_MODELS\gemma-4-26B-A4B-it-speculator.eagle3" `

  --outtype f16 `

  --target-model-dir "E:\OLLAMA_MODELS\gemma-4-26B-A4B-it" `

  --outfile "E:\OLLAMA_MODELS\gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf"

```

## Quantization

The F16 GGUF can be quantized with `llama-quantize`.

Q8_0 example:



```bash

llama-quantize \

  gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf \

  gemma-4-26B-A4B-it-speculator.eagle3-Q8_0.gguf \
  Q8_0

```



Q4_K_M example:



```bash

llama-quantize \

  gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf \

  gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf \

  Q4_K_M

```



PowerShell example:



```powershell

.\llama-quantize.exe `

  "E:\OLLAMA_MODELS\gemma-4-26B-A4B-it-speculator.eagle3-F16.gguf" `
  "E:\OLLAMA_MODELS\gemma-4-26B-A4B-it-speculator.eagle3-Q4_K_M.gguf" `

  Q4_K_M

```



## Credits



Original EAGLE3 speculator model by RedHatAI:



* `RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3`



Target GGUF model:



* `unsloth/gemma-4-26B-A4B-it-GGUF`



GGUF support and runtime:



* `ggml-org/llama.cpp`



## License



This repository is a converted GGUF version of the original speculator model. The original model license and usage terms apply. Please refer to the upstream repositories for full license details.