--- license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license thumbnail: https://huggingface.co/AtomicChat/gemma-4-31B-it-assistant-GGUF/resolve/main/hero.png base_model: - google/gemma-4-31B-it-assistant base_model_relation: quantized quantized_by: AtomicChat pipeline_tag: text-generation library_name: gguf tags: - atomic-chat - gemma - gemma4 - google - gguf - llama.cpp - quantized ---

Base model: google/gemma-4-31B-it-assistant

**Gemma 4 31B It Assistant**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Google's original weights with a per-tensor importance matrix, so this is not a repack of somebody else's files. Runs fully offline. ## Highlights - **30.7B parameters**: the weights this repo quantizes. - **Context length**: 256K tokens, as published by Google. - **60 layers**: Dense decoder, hybrid sliding-window (1024) and global attention. - **Modalities**: the base model handles Text, Image; this repo ships text-only quants, it carries no vision projector. - **Full imatrix ladder**: every quant is calibrated with an importance matrix. - **Reasoning**: All models in the family are designed as highly capable reasoners, with configurable thinking modes. - **Diverse & Efficient Architectures**: Offers Dense and Mixture-of-Experts (MoE) variants of different sizes for scalable deployment. > [!NOTE] > These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model. > [!IMPORTANT] > Always pass `--jinja` so the **Gemma 4 31B It Assistant chat template** is applied. Without it the model can emit malformed turns. ## Model Overview | Property | Value | |---|---| | Base model | `google/gemma-4-31B-it-assistant` | | Parameters | 30.7B | | Layers | 60 | | Sliding window | 1024 tokens | | Context length | 256K tokens | | Vocabulary | 262K | | Modalities | Text, Image in the base model; text only in this repo, it ships no vision projector | | Architecture | Dense decoder, hybrid sliding-window (1024) and global attention, 32 attention heads over 16 KV heads, `Gemma4AssistantForCausalLM` | | This repo | GGUF quants (imatrix). Quants: `Q4_K_S`, `Q4_K_M`, `Q5_K_M`, `Q8_0`, `F16` | ## Benchmarks | Benchmark | Score | |---|---| | MMLU Pro | 85.2% | | AIME 2026 no tools | 89.2% | | LiveCodeBench v6 | 80.0% | | Codeforces ELO | 2150 | | GPQA Diamond | 84.3% | | Tau2 (average over 3) | 76.9% | | HLE no tools | 19.5% | | HLE with search | 26.5% | | BigBench Extra Hard | 74.4% | | MMMLU | 88.4% | | MMMU Pro | 76.9% | | OmniDocBench 1.5 (average edit distance, lower is better) | 0.131 | | MATH-Vision | 85.6% | | MedXPertQA MM | 61.3% | | MRCR v2 8 needle 128k (average) | 66.4% | Scores are Google's published results for the base `google/gemma-4-31B-it-assistant`, not our own measurements. Quantization preserves the large majority of this; `Q4_K_M` and up stay close to full precision. ## Choosing a quant | Quant | Size | Notes | |---|---|---| | `Q4_K_S` | 349 MB | Compact 4-bit, fast. | | **`Q4_K_M`** | 353 MB | **Recommended default. Best balance of size, speed and quality.** | | `Q5_K_M` | 377 MB | Higher quality, low loss. | | `Q8_0` | 0.5 GB | Effectively lossless, reference quality. | | `F16` | 1.0 GB | Unquantized reference, twice the size of Q8_0. | > [!TIP] > Pick the largest file that fits your (V)RAM with room for context. `Q4_K_M` is the sweet spot for most setups; `Q6_K` or `Q8_0` for maximum fidelity. ## Get started Run Gemma 4 31B It Assistant locally with: - **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/gemma-4-31B-it-assistant-GGUF`, pick a quant, hit **Use this model**. - **llama.cpp:** `llama-server -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M --jinja -c 8192` - **Ollama:** `ollama run hf.co/AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M` - **LM Studio / Jan:** search the repo id, download any quant. ## Best practices | Parameter | Value | |---|---| | temperature | 1.0 | | top_p | 0.95 | | top_k | 64 | Google's recommended sampling configuration for `google/gemma-4-31B-it-assistant`. ## Run in llama.cpp ```bash git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server ``` ```bash ./llama.cpp/build/bin/llama-server \ -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M \ --jinja -ngl 99 -c 8192 -fa on ``` ## How these were made 1. Download `google/gemma-4-31B-it-assistant` (original weights). 2. Convert to f16 GGUF with [llama.cpp](https://github.com/ggml-org/llama.cpp). 3. Build an importance matrix over our calibration corpus. 4. Quantize the ladder with `--imatrix`. ## License Original model by Google, released under the Apache 2.0 license. Full terms: [Apache 2.0](https://ai.google.dev/gemma/docs/gemma_4_license). Quantized by Atomic Chat.