Instructions to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Use Docker

docker model run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

LM Studio
Jan
Ollama
How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Ollama:
```
ollama run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
```

Unsloth Studio

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF to start chatting

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Docker Model Runner:
```
docker model run hf.co/ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0
```

Lemonade

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Run and chat with the model

lemonade run user.gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 QAT Q4_0 MTP Assistants for ik_llama

This repo contains ik_llama-compatible GGUF assistant/draft models converted from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as --model-draft files next to the matching Google Gemma 4 QAT Q4_0 target GGUF.

Do not load this repository as the primary model. Hugging Face may show generic GGUF usage snippets for this repo, but these files are assistant/draft GGUFs only.

Important E2B/E4B Caveat

The E2B and E4B files are experimental edge-model assistants. They converted and smoke-tested successfully with matched QAT targets after an ik_llama shared-KV target loader fix.

Current ik_llama main includes that loader fix via PR #1927. Older builds may fail before draft loading with errors such as:

E2B: missing blk.15.attn_k.weight
E4B: missing blk.24.attn_k.weight

Those target tensors appear intentionally absent in Google's QAT edge GGUFs: the E2B/E4B targets advertise shared-KV tail layers.

E4B has an additional runtime caveat. On current ik_llama main, local tests could load E4B QAT + matched E4B assistant and generate in a small CPU/offload smoke path. The longer full-GPU flash-attention path was still unstable before acceptance counters in our RTX 4070 test. Treat E4B as experimental until the remaining runtime/FA path is resolved upstream.

Use matched pairs only. The E2B assistant is not expected to work with the E4B target, and the E4B assistant is not expected to work with the E2B target.

Files

Model	Q4_0 assistant
Gemma 4 E2B IT QAT Q4_0	`gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`
Gemma 4 E4B IT QAT Q4_0	`gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`
Gemma 4 12B IT QAT Q4_0	`gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`
Gemma 4 26B-A4B IT QAT Q4_0	`gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`
Gemma 4 31B IT QAT Q4_0	`gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf`

BF16 conversion intermediates and Q8_0 reference variants are intentionally not published here. The Q4_0 files are the intended pairing for Google's QAT Q4_0 target GGUFs.

Matching Target Repos

Use the assistants with Google's official target GGUFs:

google/gemma-4-E2B-it-qat-q4_0-gguf
google/gemma-4-E4B-it-qat-q4_0-gguf
google/gemma-4-12B-it-qat-q4_0-gguf
google/gemma-4-26B-A4B-it-qat-q4_0-gguf
google/gemma-4-31B-it-qat-q4_0-gguf

Example

llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja

Use a current ik_llama build with Gemma 4 MTP support. Reproducing these conversions from Google's safetensors currently requires convert_hf_to_gguf.py to recognize Gemma4UnifiedAssistantForCausalLM as the existing Gemma 4 MTP assistant converter path.

Validation

Local conversion metadata checks:

Model	Tensors	Backbone	Centroid tensors
E2B	50	1536	yes
E4B	50	2560	yes
12B	48	3840	no, metadata only
26B-A4B	48	2816	no, metadata only
31B	48	5376	no, metadata only

All published files report general.architecture=gemma4_mtp.

Historical runtime smoke on an RTX 4070 with ik_llama build 4561 (6b9de3dba):

Target + Q4_0 draft	Status	Notes
E2B	passed on shared-KV branch	MTP context ready; raw completion generated; 37/92 draft tokens accepted
E4B	passed on shared-KV branch	MTP context ready; raw completion generated; 33/116 draft tokens accepted
12B	passed	MTP context ready; raw completion generated
26B-A4B	passed	MTP context ready; raw completion generated with CPU/system-RAM offload
31B	passed	MTP context ready; raw completion generated with CPU/system-RAM offload

Additional current-main check with ik_llama bbe1a511e:

Target + Q4_0 draft	Status	Notes
E2B	passed	full-GPU flash-attention direct-server request completed
E4B	partial	small CPU/offload `-c 512` raw completion smoke passed; local full-GPU flash-attention path failed before acceptance counters

Conversion Notes

Source assistant repos:

google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant

The assistants were converted through ik_llama's Gemma 4 MTP assistant converter, then quantized with llama-quantize to Q4_0.

Downloads last month: 347

GGUF

Model size

0.4B params

Architecture

gemma4_mtp

Hardware compatibility

4-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF

Base model

google/gemma-4-12B-it-assistant

Finetuned

google/gemma-4-12B-it-qat-q4_0-unquantized-assistant

Quantized

(22)

this model