Instructions to use TheStageAI/Qwen3.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use TheStageAI/Qwen3.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheStageAI/Qwen3.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheStageAI/Qwen3.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheStageAI/Qwen3.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Ollama
How to use TheStageAI/Qwen3.5-4B-GGUF with Ollama:
```
ollama run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
```

Unsloth Studio

How to use TheStageAI/Qwen3.5-4B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting

How to use TheStageAI/Qwen3.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use TheStageAI/Qwen3.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use TheStageAI/Qwen3.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
```

Lemonade

How to use TheStageAI/Qwen3.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use TheStageAI/Qwen3.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

Qwen3.5-4B-GGUF / README.md

michellemoorre

Refine checkpoint pipeline and card layout

65a71ce verified 18 days ago

8.18 kB

license: apache-2.0
base_model:
  - Qwen/Qwen3.5-4B
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: text-generation
thumbnail: >-
  https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF/resolve/main/assets/thestage-edge-models-header.png
tags:
  - gguf
  - llama.cpp
  - quantized
  - mixed-precision
  - local-inference
  - qwen3.5

TheStageAI Edge Models — the right model at every memory budget

Qwen3.5 4B — TheStageAI GGUF

Four GGUF deployment points for llama.cpp · 1.52 GB–4.49 GB
Recommended: M · 2.39 GB · 98% of BF16 IFEval

Qwen 3.5 family: 0.8B · 2B · 4B · 9B

Start here

Tier	Size	Best for	File
XS	1.52 GB	Minimum footprint	Download
S	1.90 GB	Compact	Download
M	2.39 GB	Recommended · Balanced	Download
L	4.49 GB	Maximum fidelity	Download

Exact byte counts, SHA-256 hashes, and tensor metadata: release-manifest.json.

Run with llama.cpp

llama-cli \
  --hf-repo TheStageAI/Qwen3.5-4B-GGUF \
  --hf-file Qwen3.5-4B-M-TS-Q4_K_M.gguf

Quality

Tier	IFEval strict — prompt / instruction (%)	MMLU-Pro (%)
BF16 reference	82.44 / 87.53	79.55
XS	70.43 / 78.30	—
S	77.82 / 83.93	74.39
M	80.22 / 86.09	78.86
L	81.70 / 87.05	79.59

Modes. IFEval uses 541 deterministic non-thinking prompts. MMLU-Pro covers 12,032 questions with sampled long reasoning. Only complete MMLU-Pro runs are shown; — means not reported.

XS and reasoning: use S, M, or L for long-form reasoning. Run Qwen XS with --reasoning off.

Evaluation protocol

IFEval: 541 prompts, native chat template, enable_thinking=false, temperature 0.
MMLU-Pro: 12,032 questions, native chat template, enable_thinking=true, temperature 1, top-p 0.95, 32,768-token output limit.
The headline BF16-retention percentage uses instruction-strict IFEval and is capped at 100%; the uncapped raw scores are shown in the table.

XS targets minimum footprint for non-thinking chat and instruction following. For long-form reasoning, use S, M, or L. Run Qwen XS with --reasoning off. In matched long-thinking diagnostics, XS produced longer trajectories and reached the 32,768-token output limit more often than S. The XS MMLU-Pro cell is consequently marked —.

How the compression pipeline works

The adaptive tiers separate precision allocation from final weight reconstruction:

Q2_K … Q8_0 PTQ lanes
        ↓
per-group candidate bank + native byte costs
        ↓
ANNA-DIAG-BYTES or RCO allocation
        ↓
fresh scheduled GPTQ / QuantEase + QEP
        ↓
scale/min distillation with codes frozen

1. Build native candidates

For every quantizable tensor group, TorsionQuant produces six native K-quant candidates. Each lane uses the same Hessian-aware GPTQ stack: QuantEase optimizes the blockwise weight codes, QEP propagates reconstruction error through the layer, and native K refinement keeps the solver aligned with the GGUF layouts that will actually ship.

A uniform bank runs each Q2/Q3/Q4/Q5/Q6/Q8 lane independently.
An anchor bank builds every candidate from activations propagated along one concrete quantized trajectory.

2. Allocate the byte budget

The scheduler sees the exact native GGUF cost of every (tensor group, qtype) choice, rather than a nominal average bit width.

ANNA-DIAG-BYTES installs one candidate at a time in the BF16 model, measures full-vocabulary next-token KL against the teacher while every other group remains dense, and solves an exact multiple-choice knapsack over that isolated-loss table.
RCO optimizes the choices jointly: projected-Gumbel search minimizes end-to-end teacher KL on the interpolated model under the byte constraint, then exact integer rounding produces the hard assignment.

Adaptive tier	Bank / selector	Byte target
XS	`RCO` · anchor bank	Compact model-specific cap
S	`RCO` · anchor bank	Exact size of a pinned `UD-Q2_K_XL` reference

The S reference is used only as an external byte-budget authority.

3. Requantize and refine

The bank determines the qtype map; the release weights are regenerated from the original dense model in a fresh full-model PTQ pass. Activation propagation and error compensation therefore follow the final mixed-precision trajectory.

A short affine distillation pass then tunes only the native scales and minima against cached teacher logits. Qtypes, packed integer codes, dense weights, and tensor layouts remain frozen, so the size and runtime contract cannot drift. M and L use the same reconstruction and refinement path with fixed Q4_K and Q8_0 assignments.

Only after these stages is the final GGUF exported and measured on the untouched 3,072-sequence KL set and the downstream benchmarks above. The recommendation is based on those end-to-end results, not on the scheduler's calibration objective.

Base model: Qwen/Qwen3.5-4B at revision 851bf6e8.

Technical file details

Tier	Hub label	GGUF file type	Whole-file BPW
XS	`Q3_K_S`	`MOSTLY_Q2_K`	2.889
S	`Q4_K_S`	`MOSTLY_Q2_K`	3.622
M	`Q4_K_M`	`MOSTLY_Q4_K_M`	4.538
L	`Q8_0`	`MOSTLY_Q8_0`	8.533

For XS and S, the public Q-type suffix is an approximate Hub size label for discoverability; release-manifest.json is authoritative for tensor types. M and L use fixed Q4_K and Q8_0 assignments for quantized decoder tensors.

Runtime memory also includes KV cache and buffers, which grow with context length.

More from TheStageAI

These GGUFs provide a portable llama.cpp deployment path. For native compressed models on Apple Silicon, explore edge-lm. For custom compression, compilation, and deployment workflows, use the TheStageAI Platform and documentation.

Have a device, latency, or memory target? Talk to our team →

Reproducibility

Manifest: release-manifest.json records the exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs.
Runtime gate: export and load checks used llama.cpp revision bec4772f.

License

The model weights are released under the upstream model's Apache-2.0 license. llama.cpp and other runtime software retain their own licenses.