Instructions to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Goldkoron/Qwen3.5-397B-A17B-REAP35",
	filename="Qwen3.5-397B-A17B-REAP35-IQ2_XS_Gv2.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
llama cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
llama cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
./llama-cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Use Docker

docker model run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

LM Studio
Jan
Ollama
How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Ollama:
```
ollama run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
```

Unsloth Studio

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Docker Model Runner:
```
docker model run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
```

Lemonade

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Run and chat with the model

lemonade run user.Qwen3.5-397B-A17B-REAP35-IQ2_XS_GV

List all available models

lemonade list

Qwen3.5-397B-A17B-REAP35 / README.md

Goldkoron

Upload README.md with huggingface_hub

632a375 verified 3 months ago

2.36 kB

language:
  - en
tags:
  - gguf
  - quantized
  - moe
  - gutenberg

Qwen3.5-397B-A17B REAP35 — Gutenberg Quants

REAP35 expert-pruned (333/512 experts) quantizations of Qwen3.5-397B-A17B using the Gutenberg (Q_K_G) quantization strategy.

Available Quants

Quant	Size	BPW	Mean KLD	Same Top Token	Description
Q4_K_G	145 GiB	~4.6	0.00729	95.05%	Matches Q5_K_M quality at Q4_K_M size
Q3_K_G	117 GiB	~3.8	0.01229	93.93%	Matches Q4_K_M quality at 21% less size
IQ2_XS_G	87 GiB	~2.8	0.02922	91.20%	Beats Q3_K_M quality at 25% less size
IQ2_XXS_G	81 GiB	~2.6	0.03776	90.20%	Beats Q3_K_M quality at 30% less size

KLD measured against Q6_K reference with 32768 context, 10 chunks.

Comparison to Standard Quants

Quant	Size	Mean KLD	Same Top Token
Q5_K_M	173 GiB	0.00713	95.01%
Q4_K_G	145 GiB	0.00729	95.05%
Q4_K_M	148 GiB	0.01290	93.88%
Q3_K_G	117 GiB	0.01229	93.93%
Q3_K_M	116 GiB	0.03793	89.53%
IQ2_XS_G	87 GiB	0.02922	91.20%
Q2_K_M	89 GiB	0.10034	82.73%
IQ2_XXS_G	81 GiB	0.03776	90.20%

Q3_K_G is 3.1x better KLD than Q3_K_M at the same size. Q4_K_G matches Q5_K_M quality while being 28 GiB smaller.

What is the Gutenberg Strategy?

Gutenberg (Q_K_G) is a data-driven quantization method that allocates bit precision based on measured per-tensor KL-divergence sensitivity rather than uniform rules. A sensitivity scan identifies which tensors have the most impact on output quality, and those are preserved at higher precision while the rest are quantized aggressively. Non-expert tensors (attention, shared experts, SSM, embeddings) are kept at Q8_0 as they have disproportionate quality impact relative to their small size.

REAP Expert Pruning

These models use REAP35 pruning — 179 of 512 experts removed per layer (35% pruning) based on imatrix activation scores. This reduces model size while maintaining stable inference. REAP35 is the maximum safe pruning level for this model before quality degradation becomes noticeable.

Compatibility

Fully compatible with stock llama.cpp, llama-server, LM Studio, and any GGUF-compatible runtime. No custom builds required.