Instructions to use TheStageAI/Qwen3.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use TheStageAI/Qwen3.5-4B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use TheStageAI/Qwen3.5-4B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "TheStageAI/Qwen3.5-4B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheStageAI/Qwen3.5-4B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
- Ollama
How to use TheStageAI/Qwen3.5-4B-GGUF with Ollama:
ollama run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
- Unsloth Studio
How to use TheStageAI/Qwen3.5-4B-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for TheStageAI/Qwen3.5-4B-GGUF to start chatting
- Pi
How to use TheStageAI/Qwen3.5-4B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- OpenClaw new
How to use TheStageAI/Qwen3.5-4B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use TheStageAI/Qwen3.5-4B-GGUF with Docker Model Runner:
docker model run hf.co/TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
- Lemonade
How to use TheStageAI/Qwen3.5-4B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwen3.5-4B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use TheStageAI/Qwen3.5-4B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default TheStageAI/Qwen3.5-4B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
Refine checkpoint pipeline and card layout
Browse files- README.md +12 -7
- assets/cta-docs.svg +7 -7
- assets/cta-edge-lm.svg +8 -8
- assets/cta-platform.svg +6 -6
- release-manifest.json +5 -1
README.md
CHANGED
|
@@ -27,7 +27,7 @@ tags:
|
|
| 27 |
<strong>Recommended: M · 2.39 GB · 98% of BF16 IFEval</strong>
|
| 28 |
</p>
|
| 29 |
|
| 30 |
-
<p align="center"><a href="https://github.com/TheStageAI/edge-lm"><img src="./assets/cta-edge-lm.svg" height="36" alt="Explore edge-lm on GitHub"></a>
|
| 31 |
|
| 32 |
<p align="center"><strong>Qwen 3.5 family:</strong> <a href="https://huggingface.co/TheStageAI/Qwen3.5-0.8B-GGUF">0.8B</a> · <a href="https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF">2B</a> · <strong><a href="https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF">4B</a></strong> · <a href="https://huggingface.co/TheStageAI/Qwen3.5-9B-GGUF">9B</a></p>
|
| 33 |
|
|
@@ -75,16 +75,21 @@ XS targets minimum footprint for non-thinking chat and instruction following. Fo
|
|
| 75 |
|
| 76 |
</details>
|
| 77 |
|
| 78 |
-
## How
|
| 79 |
|
| 80 |
-
|
| 81 |
|
| 82 |
-
|
| 83 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
|
| 85 |
-
|
| 86 |
|
| 87 |
-
|
| 88 |
|
| 89 |
<details>
|
| 90 |
<summary><b>Technical file details</b></summary>
|
|
|
|
| 27 |
<strong>Recommended: M · 2.39 GB · 98% of BF16 IFEval</strong>
|
| 28 |
</p>
|
| 29 |
|
| 30 |
+
<p align="center"><a href="https://github.com/TheStageAI/edge-lm"><img src="./assets/cta-edge-lm.svg" width="104" height="36" alt="Explore edge-lm on GitHub"></a> <a href="https://docs.thestage.ai/"><img src="./assets/cta-docs.svg" width="83" height="36" alt="Read TheStageAI documentation"></a> <a href="https://app.thestage.ai/"><img src="./assets/cta-platform.svg" width="110" height="36" alt="Open TheStageAI Platform"></a></p>
|
| 31 |
|
| 32 |
<p align="center"><strong>Qwen 3.5 family:</strong> <a href="https://huggingface.co/TheStageAI/Qwen3.5-0.8B-GGUF">0.8B</a> · <a href="https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF">2B</a> · <strong><a href="https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF">4B</a></strong> · <a href="https://huggingface.co/TheStageAI/Qwen3.5-9B-GGUF">9B</a></p>
|
| 33 |
|
|
|
|
| 75 |
|
| 76 |
</details>
|
| 77 |
|
| 78 |
+
## How we build the checkpoints
|
| 79 |
|
| 80 |
+
Every tier starts from [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) at pinned revision [`851bf6e8`](https://huggingface.co/Qwen/Qwen3.5-4B/tree/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a) and passes through the same five-stage pipeline:
|
| 81 |
|
| 82 |
+
| Stage | What happens |
|
| 83 |
+
| --- | --- |
|
| 84 |
+
| **1 · Pin the source** | Lock the upstream weights, tokenizer metadata, and model topology. |
|
| 85 |
+
| **2 · Build the bank** | Produce native Q2_K–Q8_0 candidates for each quantizable tensor group and record their exact byte costs. |
|
| 86 |
+
| **3 · Select and assemble** | Choose a model-specific XS/S assignment under an exact size cap; assemble M/L with fixed Q4_K/Q8_0 assignments. |
|
| 87 |
+
| **4 · Tune scales** | Optimize representable scales and minima against teacher outputs while packed codes, dense weights, and tensor layout remain frozen. |
|
| 88 |
+
| **5 · Validate the release** | Export GGUF, verify size and payload integrity, load with pinned llama.cpp, and run held-out KL before downstream benchmarks. |
|
| 89 |
|
| 90 |
+
**Adaptive schedules for this model:** XS — `RCO anchor`; S — `RCO anchor`. `Uniform` scores module candidates independently; `anchor` uses activations from a concrete model trajectory. The S target matches the exact byte size of a pinned UD-Q2_K_XL checkpoint; its schedule and weights are produced by this pipeline.
|
| 91 |
|
| 92 |
+
The recommended tier is selected independently for each base model from complete end-to-end evaluation—not from the tier name or nominal bit label.
|
| 93 |
|
| 94 |
<details>
|
| 95 |
<summary><b>Technical file details</b></summary>
|
assets/cta-docs.svg
CHANGED
|
|
|
|
assets/cta-edge-lm.svg
CHANGED
|
|
|
|
assets/cta-platform.svg
CHANGED
|
|
|
|
release-manifest.json
CHANGED
|
@@ -36,7 +36,7 @@
|
|
| 36 |
},
|
| 37 |
"display_name": "Qwen3.5 4B",
|
| 38 |
"family": "Qwen 3.5",
|
| 39 |
-
"generated_at": "2026-07-
|
| 40 |
"license": "apache-2.0",
|
| 41 |
"model_key": "qwen3p5_4b",
|
| 42 |
"reasoning_policy": {
|
|
@@ -91,6 +91,7 @@
|
|
| 91 |
"product": "XS",
|
| 92 |
"reasoning_support": "non_thinking_only",
|
| 93 |
"recommended": false,
|
|
|
|
| 94 |
"sha256": "a1d49f1d782073748589d0df0893658040231866ae6d69667994bef5cc6ce61f",
|
| 95 |
"tensor_count": 426,
|
| 96 |
"tensor_type_counts": {
|
|
@@ -136,6 +137,7 @@
|
|
| 136 |
"product": "S",
|
| 137 |
"reasoning_support": "supported",
|
| 138 |
"recommended": false,
|
|
|
|
| 139 |
"sha256": "9f020bfc77a9503a641e3e0511de04b4bfba16b9facfa62c3772b4f8e318ba55",
|
| 140 |
"tensor_count": 426,
|
| 141 |
"tensor_type_counts": {
|
|
@@ -183,6 +185,7 @@
|
|
| 183 |
"product": "M",
|
| 184 |
"reasoning_support": "supported",
|
| 185 |
"recommended": true,
|
|
|
|
| 186 |
"sha256": "f8e45572b9cc35161d4772b09bccfd383fe0bb03fc6d69b40a9138731302290b",
|
| 187 |
"tensor_count": 426,
|
| 188 |
"tensor_type_counts": {
|
|
@@ -226,6 +229,7 @@
|
|
| 226 |
"product": "L",
|
| 227 |
"reasoning_support": "supported",
|
| 228 |
"recommended": false,
|
|
|
|
| 229 |
"sha256": "b81d86c3ccbd1af44474597c0ef97af8e4d71ba40f3b5e7a20b4d6c2f709203c",
|
| 230 |
"tensor_count": 426,
|
| 231 |
"tensor_type_counts": {
|
|
|
|
| 36 |
},
|
| 37 |
"display_name": "Qwen3.5 4B",
|
| 38 |
"family": "Qwen 3.5",
|
| 39 |
+
"generated_at": "2026-07-21T10:34:15.339530+00:00",
|
| 40 |
"license": "apache-2.0",
|
| 41 |
"model_key": "qwen3p5_4b",
|
| 42 |
"reasoning_policy": {
|
|
|
|
| 91 |
"product": "XS",
|
| 92 |
"reasoning_support": "non_thinking_only",
|
| 93 |
"recommended": false,
|
| 94 |
+
"schedule_method": "RCO anchor",
|
| 95 |
"sha256": "a1d49f1d782073748589d0df0893658040231866ae6d69667994bef5cc6ce61f",
|
| 96 |
"tensor_count": 426,
|
| 97 |
"tensor_type_counts": {
|
|
|
|
| 137 |
"product": "S",
|
| 138 |
"reasoning_support": "supported",
|
| 139 |
"recommended": false,
|
| 140 |
+
"schedule_method": "RCO anchor",
|
| 141 |
"sha256": "9f020bfc77a9503a641e3e0511de04b4bfba16b9facfa62c3772b4f8e318ba55",
|
| 142 |
"tensor_count": 426,
|
| 143 |
"tensor_type_counts": {
|
|
|
|
| 185 |
"product": "M",
|
| 186 |
"reasoning_support": "supported",
|
| 187 |
"recommended": true,
|
| 188 |
+
"schedule_method": "fixed Q4_K",
|
| 189 |
"sha256": "f8e45572b9cc35161d4772b09bccfd383fe0bb03fc6d69b40a9138731302290b",
|
| 190 |
"tensor_count": 426,
|
| 191 |
"tensor_type_counts": {
|
|
|
|
| 229 |
"product": "L",
|
| 230 |
"reasoning_support": "supported",
|
| 231 |
"recommended": false,
|
| 232 |
+
"schedule_method": "fixed Q8_0",
|
| 233 |
"sha256": "b81d86c3ccbd1af44474597c0ef97af8e4d71ba40f3b5e7a20b4d6c2f709203c",
|
| 234 |
"tensor_count": 426,
|
| 235 |
"tensor_type_counts": {
|