Instructions to use AtomicChat/gemma-4-31B-it-assistant-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Use Docker

docker model run hf.co/AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AtomicChat/gemma-4-31B-it-assistant-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AtomicChat/gemma-4-31B-it-assistant-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Ollama
How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with Ollama:
```
ollama run hf.co/AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
```

Unsloth Studio

How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AtomicChat/gemma-4-31B-it-assistant-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AtomicChat/gemma-4-31B-it-assistant-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for AtomicChat/gemma-4-31B-it-assistant-GGUF to start chatting

Docker Model Runner
How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with Docker Model Runner:
```
docker model run hf.co/AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M
```

Lemonade

How to use AtomicChat/gemma-4-31B-it-assistant-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AtomicChat/gemma-4-31B-it-assistant-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-31B-it-assistant-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Your github repo with the recommended settings you listed on this huggingface repo fail with Gemma 4 31B on Windows 11 RTX 3090

by noodlecake - opened May 9

Discussion

noodlecake

May 9

•

edited May 9

Your github repo with the recommended settings you listed on this huggingface repo fail with Gemma 4 31B on Windows 11 RTX 3090 upon sending a message to the server.

"E:\AI\llama-cpp\atomic-llama-cpp-turboquant\build\bin\Release\llama-server.exe" -m "E:\AI\Comfy_Models\models\LLM\GEMMA 4\gemma-4-31B-it-UD-Q4_K_XL.gguf" --mtp-head "E:\AI\Comfy_Models\models\LLM\GEMMA 4\Draft Models\g4atomic-4-31B-it-assistant.Q8_0.gguf" --spec-type mtp --draft-block-size 3 --draft-max 8 --draft-min 0 -ngl 99 -ngld 99 -ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3 -fa on -c 16384 --host 127.0.0.1 --port 8080
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b8996-2374b99c1
system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model 'E:\AI\Comfy_Models\models\LLM\GEMMA 4\gemma-4-31B-it-UD-Q4_K_XL.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 19411 MiB of device memory vs. 23335 MiB of free device memory
llama_params_fit_impl: will leave 3923 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.81 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:07:00.0) - 23335 MiB free
llama_model_loader: loaded meta data with 57 key-value pairs and 833 tensors from E:\AI\Comfy_Models\models\LLM\GEMMA 4\gemma-4-31B-it-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = gemma4
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 64
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 5: general.name str = Gemma-4-31B-It
llama_model_loader: - kv 6: general.finetune str = it
llama_model_loader: - kv 7: general.basename str = Gemma-4-31B-It
llama_model_loader: - kv 8: general.quantized_by str = Unsloth
llama_model_loader: - kv 9: general.size_label str = 31B
llama_model_loader: - kv 10: general.license str = apache-2.0
llama_model_loader: - kv 11: general.license.link str = https://ai.google.dev/gemma/docs/gemm...
llama_model_loader: - kv 12: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 13: general.base_model.count u32 = 1
llama_model_loader: - kv 14: general.base_model.0.name str = Gemma 4 31B It
llama_model_loader: - kv 15: general.base_model.0.organization str = Google
llama_model_loader: - kv 16: general.base_model.0.repo_url str = https://huggingface.co/google/gemma-4...
llama_model_loader: - kv 17: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv 18: gemma4.block_count u32 = 60
llama_model_loader: - kv 19: gemma4.context_length u32 = 262144
llama_model_loader: - kv 20: gemma4.embedding_length u32 = 5376
llama_model_loader: - kv 21: gemma4.feed_forward_length u32 = 21504
llama_model_loader: - kv 22: gemma4.attention.head_count u32 = 32
llama_model_loader: - kv 23: gemma4.attention.head_count_kv arr[i32,60] = [16, 16, 16, 16, 16, 4, 16, 16, 16, 1...
llama_model_loader: - kv 24: gemma4.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 25: gemma4.rope.freq_base_swa f32 = 10000.000000
llama_model_loader: - kv 26: gemma4.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 27: gemma4.attention.key_length u32 = 512
llama_model_loader: - kv 28: gemma4.attention.value_length u32 = 512
llama_model_loader: - kv 29: gemma4.final_logit_softcapping f32 = 30.000000
llama_model_loader: - kv 30: gemma4.attention.sliding_window u32 = 1024
llama_model_loader: - kv 31: gemma4.attention.shared_kv_layers u32 = 0
llama_model_loader: - kv 32: gemma4.embedding_length_per_layer_input u32 = 0
llama_model_loader: - kv 33: gemma4.attention.sliding_window_pattern arr[bool,60] = [true, true, true, true, true, false,...
llama_model_loader: - kv 34: gemma4.attention.key_length_swa u32 = 256
llama_model_loader: - kv 35: gemma4.attention.value_length_swa u32 = 256
llama_model_loader: - kv 36: gemma4.rope.dimension_count u32 = 512
llama_model_loader: - kv 37: gemma4.rope.dimension_count_swa u32 = 256
llama_model_loader: - kv 38: tokenizer.ggml.model str = gemma4
llama_model_loader: - kv 39: tokenizer.ggml.tokens arr[str,262144] = ["", "", "", "", ...
llama_model_loader: - kv 40: tokenizer.ggml.scores arr[f32,262144] = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv 41: tokenizer.ggml.token_type arr[i32,262144] = [3, 1, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 42: tokenizer.ggml.merges arr[str,514906] = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv 43: tokenizer.ggml.bos_token_id u32 = 2
llama_model_loader: - kv 44: tokenizer.ggml.eos_token_id u32 = 106
llama_model_loader: - kv 45: tokenizer.ggml.unknown_token_id u32 = 3
llama_model_loader: - kv 46: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 47: tokenizer.ggml.mask_token_id u32 = 4
llama_model_loader: - kv 48: tokenizer.chat_template str = {%- macro format_parameters(propertie...
llama_model_loader: - kv 49: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 50: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 51: general.quantization_version u32 = 2
llama_model_loader: - kv 52: general.file_type u32 = 15
llama_model_loader: - kv 53: quantize.imatrix.file str = gemma-4-31B-it-GGUF/imatrix_unsloth.gguf
llama_model_loader: - kv 54: quantize.imatrix.dataset str = unsloth_calibration_gemma-4-31B-it.txt
llama_model_loader: - kv 55: quantize.imatrix.entries_count u32 = 410
llama_model_loader: - kv 56: quantize.imatrix.chunks_count u32 = 141
llama_model_loader: - type f32: 422 tensors
llama_model_loader: - type q4_K: 304 tensors
llama_model_loader: - type q5_K: 37 tensors
llama_model_loader: - type q6_K: 70 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 17.52 GiB (4.90 BPW)
load: 0 unused tokens
load: control-looking token: 1 '' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load: - 1 ('')
load: - 50 ('<|tool_response>')
load: - 106 ('<turn|>')
load: - 212 ('')
load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
load: special tokens cache size = 24
load: token to piece cache size = 1.9445 MB
print_info: arch = gemma4
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 5376
print_info: n_embd_inp = 5376
print_info: n_layer = 60
print_info: n_head = 32
print_info: n_head_kv = [16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4, 16, 16, 16, 16, 16, 4]
print_info: n_rot = 512
print_info: n_swa = 1024
print_info: is_swa_any = 1
print_info: n_embd_head_k = 512
print_info: n_embd_head_v = 512
print_info: n_gqa = [2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8]
print_info: n_embd_k_gqa = [4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048]
print_info: n_embd_v_gqa = [4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048, 4096, 4096, 4096, 4096, 4096, 2048]
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 1.0e+00
print_info: n_ff = 21504
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: freq_base_swa = 10000.0
print_info: freq_scale_swa = 1
print_info: n_embd_head_k_swa = 256
print_info: n_embd_head_v_swa = 256
print_info: n_rot_swa = 256
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: model type = ?B
print_info: model params = 30.70 B
print_info: general.name = Gemma-4-31B-It
print_info: vocab type = BPE
print_info: n_vocab = 262144
print_info: n_merges = 514906
print_info: BOS token = 2 ''
print_info: EOS token = 106 '<turn|>'
print_info: UNK token = 3 ''
print_info: PAD token = 0 ''
print_info: MASK token = 4 '

C:\Users\noodl>

sujitvasanth

May 11

•

edited May 11

hi i;ve tested this extensively and runs on my rts3090 on ubuntu20.04 just fine... your log isnt showing any errors, is it truncated?
suggest try trubo2 (lower vram) and a smaller model I like unsloth's UD.iQ3_XXS... even on my 18gb other setup i can have context of 250k so i think your context is ok
also you have to have the right architecture for your gemma draft model gguf - you need to use the versionin this repo... other draft model versions generated with different gguf creators dont have the right architecture and will crash

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment