Instructions to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Goldkoron/Qwen3.5-397B-A17B-REAP35",
	filename="Qwen3.5-397B-A17B-REAP35-IQ2_XS_Gv2.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
llama cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
llama cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
./llama-cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Use Docker

docker model run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

LM Studio
Jan
Ollama
How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Ollama:
```
ollama run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
```

Unsloth Studio

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Goldkoron/Qwen3.5-397B-A17B-REAP35 to start chatting

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Docker Model Runner:
```
docker model run hf.co/Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV
```

Lemonade

How to use Goldkoron/Qwen3.5-397B-A17B-REAP35 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Goldkoron/Qwen3.5-397B-A17B-REAP35:IQ2_XS_GV

Run and chat with the model

lemonade run user.Qwen3.5-397B-A17B-REAP35-IQ2_XS_GV

List all available models

lemonade list

Note: Download these only to experiment, I personally believe deep quantizations of the unreaped model are better and more productive. REAP seems to break things in the routing that I don't fundamentally understand.

Qwen3.5-397B-A17B REAP35 — Gutenberg Quants

REAP35 expert-pruned (333/512 experts) quantizations of Qwen3.5-397B-A17B using the Gutenberg (Q_K_G) quantization strategy.

Available Quants

Quant	Size	BPW	Mean KLD	Same Top Token	Description
Q8_0	258 GiB	8.51	—	—	Source model, maximum quality
Q4_K_G	148 GiB	4.86	0.00751	94.26%	Approaches Q5_K_M quality at Q4_K_M size
Q3_K_G	116 GiB	3.83	0.00932	94.68%	Beats Q4_K_M quality at 22% less size
IQ2_XS_G	87 GiB	2.86	0.02150	92.55%	Beats Q3_K_M quality at 25% less size

Comparison to Standard Quants

Quant	Size	BPW	Mean KLD	Same Top Token
Q5_K_M	173 GiB	5.69	0.00642	95.18%
Q4_K_G	148 GiB	4.86	0.00751	94.26%
Q4_K_M	148 GiB	4.86	0.01242	93.67%
Q3_K_G	116 GiB	3.83	0.00932	94.68%
Q3_K_M	116 GiB	3.83	0.03797	89.36%
IQ2_XS_G	87 GiB	2.86	0.02150	92.55%
Q2_K	89 GiB	2.93	0.10118	82.63%

Q3_K_G at 116 GiB beats Q4_K_M at 148 GiB — better quality, 22% smaller
Q4_K_G is 1.7x better KLD than Q4_K_M at the same size
IQ2_XS_G is 4.7x better KLD than Q2_K at a smaller size

Gutenberg Quantization

Gutenberg (Q_K_G) is a data-driven quantization method. A KLD sensitivity scan measures each expert tensor's impact on output quality, and tensors are ranked by importance. The most impactful tensors receive higher precision while the rest are quantized at the base level. Non-expert tensors are kept at Q8_0 for their disproportionate quality impact.

REAP Expert Pruning

REAP scores each expert using imatrix calibration data and uniformly removes the lowest-scoring experts from every MoE layer.

Each expert receives a score based on two signals captured during calibration inference:

Activation count — how many times the expert was selected by the router
Activation magnitude — sum of squared input activations when the expert was active

The final score is: normalized_count x normalized_magnitude

Base model: Qwen3.5-397B-A17B (512 experts, 10 active per layer)
Pruned: 512 → 333 experts per layer (35% removed)

Compatibility

Fully compatible with stock llama.cpp, llama-server, LM Studio, and any GGUF-compatible runtime. No custom builds required.

Downloads last month: 45

GGUF

Model size

261B params

Architecture

qwen35moe

Hardware compatibility

2-bit

8-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Goldkoron/Qwen3.5-397B-A17B-REAP35

Base model

Qwen/Qwen3.5-397B-A17B

Quantized

(67)

this model