Instructions to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Use Docker

docker model run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

LM Studio
Jan
Ollama
How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Ollama:
```
ollama run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
```

Unsloth Studio

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Docker Model Runner:
```
docker model run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
```

Lemonade

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Run and chat with the model

lemonade run user.gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF / README.md

ji-farthing

Add experimental E2B/E4B assistants

d8381c5 verified 3 months ago

4.7 kB

license: apache-2.0
base_model:
  - google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-31B-it-qat-q4_0-unquantized-assistant
tags:
  - gguf
  - gemma4
  - qat
  - q4_0
  - speculative-decoding
  - mtp
  - ik_llama

Gemma 4 QAT Q4_0 MTP Assistants for ik_llama

This repo contains ik_llama-compatible GGUF assistant/draft models converted from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as --model-draft files next to the matching Google Gemma 4 QAT Q4_0 target GGUF.

Important E2B/E4B Caveat

The E2B and E4B files are experimental edge-model assistants. They converted and smoke-tested successfully with matched QAT targets only after an ik_llama shared-KV target loader fix.

Current ik_llama builds without that fix may fail before draft loading with errors such as:

E2B: missing blk.15.attn_k.weight
E4B: missing blk.24.attn_k.weight

Those target tensors appear intentionally absent in Google's QAT edge GGUFs: the E2B/E4B targets advertise shared-KV tail layers. A test branch with the loader fix is here:

https://github.com/joelfarthing/ik_llama.cpp/tree/filament/gemma4-edge-shared-kv-20260606

Use matched pairs only. The E2B assistant is not expected to work with the E4B target, and the E4B assistant is not expected to work with the E2B target.

Files

Model	Q4_0 assistant	Q8_0 assistant
Gemma 4 E2B IT QAT Q4_0	`gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`	`gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf`
Gemma 4 E4B IT QAT Q4_0	`gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`	`gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf`
Gemma 4 12B IT QAT Q4_0	`gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`	`gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf`
Gemma 4 26B-A4B IT QAT Q4_0	`gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`	`gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf`
Gemma 4 31B IT QAT Q4_0	`gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`	`gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf`

BF16 conversion intermediates are not published here.

Matching Target Repos

Use the assistants with Google's official target GGUFs:

google/gemma-4-E2B-it-qat-q4_0-gguf
google/gemma-4-E4B-it-qat-q4_0-gguf
google/gemma-4-12B-it-qat-q4_0-gguf
google/gemma-4-26B-A4B-it-qat-q4_0-gguf
google/gemma-4-31B-it-qat-q4_0-gguf

Example

llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja

Use a current ik_llama build with Gemma 4 MTP support. Reproducing these conversions from Google's safetensors currently requires convert_hf_to_gguf.py to recognize Gemma4UnifiedAssistantForCausalLM as the existing Gemma 4 MTP assistant converter path.

Validation

Local conversion metadata checks:

Model	Tensors	Backbone	Centroid tensors
E2B	50	1536	yes
E4B	50	2560	yes
12B	48	3840	no, metadata only
26B-A4B	48	2816	no, metadata only
31B	48	5376	no, metadata only

All published files report general.architecture=gemma4_mtp.

Runtime smoke on an RTX 4070 with ik_llama build 4561 (6b9de3dba):

Target + Q4_0 draft	Status	Notes
E2B	passed on shared-KV branch	MTP context ready; raw completion generated; 37/92 draft tokens accepted
E4B	passed on shared-KV branch	MTP context ready; raw completion generated; 33/116 draft tokens accepted
12B	passed	MTP context ready; raw completion generated
26B-A4B	passed	MTP context ready; raw completion generated with CPU/system-RAM offload
31B	passed	MTP context ready; raw completion generated with CPU/system-RAM offload

Conversion Notes

Source assistant repos:

google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant

The assistants were converted through ik_llama's Gemma 4 MTP assistant converter, then quantized with llama-quantize to Q4_0 and Q8_0.

The Q4_0 files are the intended pairing for the QAT Q4_0 targets. The Q8_0 files are provided as higher-fidelity draft/reference variants.