Instructions to use ubergarm/Qwen3.5-122B-A10B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Use Docker

docker model run hf.co/ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

LM Studio
Jan

vLLM

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/Qwen3.5-122B-A10B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/Qwen3.5-122B-A10B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Ollama
How to use ubergarm/Qwen3.5-122B-A10B-GGUF with Ollama:
```
ollama run hf.co/ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
```
Unsloth Desktop

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ubergarm/Qwen3.5-122B-A10B-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K
```

Lemonade

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Run and chat with the model

lemonade run user.Qwen3.5-122B-A10B-GGUF-Q2_K

List all available models

lemonade list

Hermes Agent

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ubergarm/Qwen3.5-122B-A10B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ubergarm/Qwen3.5-122B-A10B-GGUF:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Missing about 50~55GB of Q3?

by lingyezhixing - opened Feb 26

Discussion

lingyezhixing

Feb 26

Now there are ~35GB, ~45GB, ~65GB, but not 55GB

ubergarm

Owner Feb 26

啊哈，是的，我想我明白了。你想要一个大概能放入 55GiB 内存 + 显存的量化模型是吗？我猜你用的是 ik_llama.cpp 对吧？另外看看新的 -fdn 512 功能，它通过融合门控 delta 网络来加速 qwen35moe 和 qwe3next，以获得更快的 token 生成速度。

那我会考虑添加另一个在 64GB 总内存 + 显存下表现良好的量化版本。最近忙着处理这么多新模型，而且我在这里的公共配额有限。你知道人们还在哪里能找到我的模型吗？

谢谢！

此评论由 ubergarm/Qwen3.5-397B-A17B Q3_K 翻译

Ahh yes, I believe I understand. You would like to have a quantized model that fits in roughly 55GiB of RAM+VRAM? I assume you are using ik_llama.cpp yes? Also check out the new -fdn 512 feature that speeds up qwen35moe and qwe3next by fusing the gated delta net for more token generation speed.

I'll consider adding another quant that will work well in 64GB total RAM+VRAM then. It has been busy with so many new models, and I have limited public quota here. Where else do people find my models do you know?

Thanks!

This comment translated by ubergarm/Qwen3.5-397B-A17B Q3_K

lingyezhixing

Feb 26

It's not necessary at all; I'm just tentatively offering a suggestion rather than demanding it, please don't feel any responsibility or pressure

Regarding where to find your model, I mostly use Hugging Face, and sometimes I also use ModelScope (I hope I haven't misunderstood the question)

watchingyousleep

Mar 3

I don't know I'd use it over the Unsloth UD-Q3_K_XL but for what its worth I do fall in the 55GB range and would have downloaded the model for testing purposes.

igor255

Mar 13

I made my own quant from BF16, i like it so far.
Its 53gb in size and has sub 3% perplexity.

ffn_gate_exps.weight: IQ3_KS: 48 tensors (38.65B elements)
ffn_up_exps.weight: IQ3_KS: 48 tensors (38.65B elements)
ffn_down_exps.weight: IQ3_KS: 48 tensors (38.65B elements)
attn_qkv.weight: Q8_0: 36 tensors (1.36B elements)
attn_gate.weight: Q8_0: 36 tensors (905.97M elements)
ssm_out.weight: Q8_0: 36 tensors (905.97M elements)
output.weight: Q8_0: 1 tensors (762.84M elements)
token_embd.weight: Q8_0: 1 tensors (762.84M elements)
attn_q.weight: Q8_0: 12 tensors (603.98M elements)
attn_output.weight: Q8_0: 12 tensors (301.99M elements)
ffn_down_shexp.weight: Q8_0: 48 tensors (150.99M elements)
ffn_gate_shexp.weight: Q8_0: 48 tensors (150.99M elements)
ffn_up_shexp.weight: Q8_0: 48 tensors (150.99M elements)
ffn_gate_inp.weight: F32: 48 tensors (37.75M elements)
attn_k.weight: Q8_0: 12 tensors (18.87M elements)
attn_v.weight: Q8_0: 12 tensors (18.87M elements)
ssm_alpha.weight: BF16: 36 tensors (7.08M elements)
ssm_beta.weight: BF16: 36 tensors (7.08M elements)
// Rest are small vectors in F32

ubergarm

Owner Mar 13

@igor255

Nice! Looks like a solid recipe, that looks exactly like what I'd make for a smol-IQ3_KS!

Yes the trend seems to be to keep those ssm guys at full quality bf16 (or f32 if your specific backend is actually faster with that?) (but don't downcast it to f16 as it could cause clipping). I'm not sure where the original research is suggesting that even q8_0 effects long context performance, but I'm still not sure about that, but it isn't a lot of size difference so seems fine.

Feel free to upload it to HF, if you need any guidance I just shared some tips with @tarruda here: https://www.reddit.com/r/LocalLLaMA/comments/1rotwhr/comment/o9uxvlc/ as he just uploaded a new option for the big size here: https://huggingface.co/tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF

Cheers and happy weekend y'all!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment