Instructions to use mradermacher/gemma-4-31B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/gemma-4-31B-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/gemma-4-31B-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/gemma-4-31B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/gemma-4-31B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/gemma-4-31B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/gemma-4-31B-GGUF with Ollama:
```
ollama run hf.co/mradermacher/gemma-4-31B-GGUF:Q4_K_M
```

Unsloth Studio

How to use mradermacher/gemma-4-31B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/gemma-4-31B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/gemma-4-31B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/gemma-4-31B-GGUF to start chatting

Docker Model Runner
How to use mradermacher/gemma-4-31B-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/gemma-4-31B-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/gemma-4-31B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/gemma-4-31B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-31B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Please upload BF16 version of this model in GGUF?

by luxiloid - opened Apr 11

Discussion

luxiloid

Apr 11

I have hard time converting it to BF16 gguf myself. Hope you could upload it.

nicoboss

Apr 14

•

edited Apr 26

There is absolutely no benefit running this model in BF16 over Q8_0. Everything i1-Q5_K_M and larger will not result in any quality difference that can be perceived by humans in a monolithic model based on the measurements I made in Q4 2024. If you absolutely insist on us providing BF16 quants anyways we could do so.

87455a

Apr 25

"larger will not result in any quality difference that can be perceived by pajeet"
Fixed it for you.

nicoboss

Apr 26

If we look at percentage compared to BF16 for Qwen2.5-32B & Qwen2.5-32B-Instruct I did back then you can see that Q8 and even i1-Q6 booth have 100% correct token probability and under evel even better due to not even thousands of benchmarks results enough accuracy to tell a difference between them. With 99.86% perplexity and 99.72% KL-divergence the token probabilities are almost identical. 97.01% same token probability shows that sometimes the second-best token wins in 3% of cases but any randomness added by temperature far exceeds any quality difference caused by going from BF16 to Q8. The floating-point rounding inaccuracies alone between different inference engine might cause a similar difference in same token probability but that's just speculation as I never measured it. In any case at least in my opinion the results clearly show why its not worth it to ever use BF16 quants. They just make the model waste way more resources (GPU memory and GPU bandwidth) which usualy results in slower inference speed at higher power consumption for no real benefit to any human user. The only reason I see to use BF16 would be some benchmark with hundred thousands of questions and even then the difference will be so tiny that unless the benchmark is truly massive you won't be able to tell any difference.

Qwen2.5-32B & Qwen2.5-32B-Instruct

Quant	Rank	KL-divergence	Correct token	Same token	Perplexity	Eval
Q8_0	1	99.72	100.05	97.01	99.86	100.38
i1-Q6_K	2	99.42	100.00	95.99	99.72	100.77
Q6_K	3	99.38	99.98	95.88	99.60	100.70
i1-Q5_K_M	4	98.76	99.88	94.81	99.11	100.29
i1-Q5_K_S	5	98.61	99.89	94.61	99.04	100.17
i1-Q5_1	6	98.69	99.85	94.73	98.97	100.17
i1-Q5_0	7	98.45	99.92	94.39	98.71	100.33
Q5_K_M	8	98.62	99.84	94.56	99.15	100.68
Q5_K_S	9	98.35	99.79	94.15	99.02	100.78
Q5_0	10	98.12	99.80	93.86	98.37	100.25
Q5_1	11	98.23	99.76	93.95	98.78	100.00
i1-Q4_K_M	12	96.76	99.65	92.52	97.98	100.33
i1-IQ4_NL	13	96.13	99.68	91.88	97.67	100.80
i1-Q4_K_S	14	96.23	99.57	92.06	97.64	100.81
i1-Q4_1	15	96.28	99.53	92.07	97.60	99.86
i1-IQ4_XS	16	96.08	99.68	91.86	97.59	100.30
Q4_K_M	17	96.34	99.45	92.02	97.39	99.86
IQ4_NL	18	95.63	99.43	91.26	97.43	100.61
IQ4_XS	19	95.54	99.43	91.10	97.41	100.23
Q4_K_S	20	95.61	99.29	91.24	97.21	100.39
i1-Q4_0	21	94.95	99.30	90.73	96.83	101.10
Q4_1	22	94.59	98.99	90.37	96.68	100.62
Q4_0	23	93.95	99.01	89.78	96.10	99.66
i1-Q3_K_L	24	92.02	98.94	88.87	95.02	101.33
i1-Q3_K_M	25	91.04	98.83	88.25	94.36	100.47
Q3_K_L	26	90.61	98.35	87.79	93.72	100.04
Q3_K_M	27	89.18	98.02	86.84	92.70	99.33
i1-IQ3_S	28	88.82	97.86	86.80	91.44	99.10
i1-IQ3_M	29	88.79	97.68	86.80	91.12	99.26
i1-IQ3_XS	30	86.52	97.64	85.62	90.24	100.26
i1-Q3_K_S	31	85.81	97.48	84.60	89.77	101.30
Q3_K_S	32	84.02	97.05	83.70	87.78	99.08
i1-IQ3_XXS	33	82.46	97.23	83.37	87.10	100.64
IQ3_M	34	78.93	96.12	81.08	81.98	96.38
IQ3_S	35	76.56	96.17	80.20	80.53	97.68
i1-Q2_K	36	74.42	95.98	79.69	79.69	99.42
IQ3_XS	37	74.06	95.73	79.22	78.69	98.12
i1-IQ2_M	38	71.61	95.50	78.65	76.56	99.59
i1-Q2_K_S	39	68.24	95.59	78.06	72.98	97.17
Q2_K	40	66.38	94.33	76.91	69.93	96.08
i1-IQ2_S	41	63.37	94.26	75.83	68.29	98.92
i1-IQ2_XS	42	61.16	93.79	75.23	65.91	95.93
i1-IQ2_XXS	43	51.61	92.11	72.35	53.91	95.04
i1-IQ1_M	44	24.26	87.08	64.29	10.70	87.40
i1-IQ1_S	45	5.82	82.99	60.06	-26.44	80.70

RichardErkhov

Apr 26

i1-IQ1_S 45 5.82 82.99 60.06 -26.44 80.70

that looks so wrong lmao

87455a

Apr 28

Wow, that's a lot of numbers and big words. Too bad I'm not reading them.
I can simply run a model at Q5, see that it constantly confuses a buttcrack for a boob cleavage, run it at BF16, and the problem magically disappears.
I am on another, unfathomable, level that you will never achieve.
Your cope for justifying not buying more vram shall only hold you back.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment