Instructions to use TheStageAI/Qwen3.5-2B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use TheStageAI/Qwen3.5-2B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheStageAI/Qwen3.5-2B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheStageAI/Qwen3.5-2B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheStageAI/Qwen3.5-2B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Ollama
How to use TheStageAI/Qwen3.5-2B-GGUF with Ollama:
```
ollama run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
```

Unsloth Studio

How to use TheStageAI/Qwen3.5-2B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

How to use TheStageAI/Qwen3.5-2B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use TheStageAI/Qwen3.5-2B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use TheStageAI/Qwen3.5-2B-GGUF with Docker Model Runner:
```
docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
```

Lemonade

How to use TheStageAI/Qwen3.5-2B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-2B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use TheStageAI/Qwen3.5-2B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

michellemoorre commited on 18 days ago

Commit

9bdfbd9

verified ·

1 Parent(s): accf1d9

Update private release-card draft v2

Browse files

Files changed (2) hide show

README.md +54 -27
release-manifest.json +22 -5

README.md CHANGED Viewed

@@ -14,60 +14,87 @@ tags:
 # Qwen3.5 2B — TheStageAI GGUF
-Four text-only GGUF checkpoints from **0.738 GB** to **2.013 GB**, evaluated on IFEval and MMLU-Pro where a complete release score is available.
-[Qwen3.5 0.8B](https://huggingface.co/TheStageAI/Qwen3.5-0.8B-GGUF) · **[Qwen3.5 2B](https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF)** · [Qwen3.5 4B](https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF) · [Qwen3.5 9B](https://huggingface.co/TheStageAI/Qwen3.5-9B-GGUF) · [Gemma 4 E2B IT](https://huggingface.co/TheStageAI/gemma-4-E2B-it-GGUF) · [Gemma 4 E4B IT](https://huggingface.co/TheStageAI/gemma-4-E4B-it-GGUF) · [Gemma 4 12B IT](https://huggingface.co/TheStageAI/gemma-4-12B-it-GGUF)
-## Choose a file
-| Variant | Hub class | File | Size | Whole-file BPW | IFEval P / I (%) | MMLU-Pro (%) | Use |
-| --- | --- | --- | ---: | ---: | ---: | ---: | --- |
-| XS | `Q3_K_S` | `Qwen3.5-2B-XS-TS-Q3_K_S.gguf` | 0.738 GB | 3.140 | 52.68 / 64.15 | — | Minimum size |
-| S | `Q4_K_S` | `Qwen3.5-2B-S-TS-Q4_K_S.gguf` | 0.967 GB | 4.109 | 63.22 / 73.38 | — | Compact |
-| **M** | `Q4_K_M` | `Qwen3.5-2B-M-TS-Q4_K_M.gguf` | 1.073 GB | 4.562 | 66.54 / 75.54 | — | **Recommended · Uniform Q4** |
-| L | `Q8_0` | `Qwen3.5-2B-L-TS-Q8_0.gguf` | 2.013 GB | 8.558 | 65.80 / 74.94 | — | Uniform Q8 |
-> **Hub class versus file type:** XS and S are TheStage mixed-precision schedules. Their Hub labels describe the whole-file size class for discoverability; they are not stock `Q3_K_S`, `Q3_K_L`, or `Q4_K_S` conversions. M and L use the actual `MOSTLY_Q4_K_M` and `MOSTLY_Q8_0` GGUF file types. Exact tensor-type counts and SHA-256 digests are in [`release-manifest.json`](./release-manifest.json).
-IFEval is shown as prompt-strict / instruction-strict. The BF16 reference scored **65.43 / 74.70** on IFEval. MMLU-Pro is not reported for the BF16 reference in this release.
 ## Quickstart
-Use a current [llama.cpp](https://github.com/ggml-org/llama.cpp) build. The command below selects the recommended **M** file explicitly through its Hub quant label:
 ```bash
-llama-cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
 ```
-## Evaluation
 - **IFEval:** 541 prompts, native chat template, `enable_thinking=false`, temperature 0.
-- **MMLU-Pro:** 12,032 questions for complete rows, native chat template, `enable_thinking=true`, temperature 1, top-p 0.95.
-- A dash means **not reported**, not zero and not a score reconstructed from partial subjects.
-These protocols intentionally exercise different operating modes. IFEval measures non-thinking instruction and format adherence; MMLU-Pro includes long sampled reasoning.
-## What this release is
-This release starts from [`Qwen/Qwen3.5-2B`](https://huggingface.co/Qwen/Qwen3.5-2B) at revision [`15852e8c16360a2fea060d615a32b45270f8a8fc`](https://huggingface.co/Qwen/Qwen3.5-2B/tree/15852e8c16360a2fea060d615a32b45270f8a8fc) and applies TheStageAI's scale-tuned deployment compression pipeline.
-- **XS:** minimum-size adaptive mixed precision.
 - **S:** compact adaptive mixed precision.
-- **M:** uniform Q4 operating point.
-- **L:** uniform Q8 operating point.
-The four products are operating points, not a promise that benchmark scores increase monotonically with file size. The recommended row is selected separately for each base model from the release evaluations.
 ## Limitations
-- The files in this repository contain the language-model GGUF. Multimodal projector files are not included.
 - Small score reversals between BF16 and quantized rows should be read as evaluation variation, not as a claim that quantization improves the base model.
-- For XS, use `enable_thinking=false`. A headline thinking-mode MMLU-Pro score is not reported when long generations do not produce a stable product metric.
-- MMLU-Pro is not reported for this model in this release; diagnostic partial subjects are deliberately not aggregated.
 ## Provenance
-The exact base revision, file sizes, GGUF file types, whole-file BPW, SHA-256 digests, held-out KL values, and evaluation IDs are recorded in [`release-manifest.json`](./release-manifest.json). Release filenames differ from the internal artifact paths; the payload bytes must match those digests exactly.
 ## License

 # Qwen3.5 2B — TheStageAI GGUF
+Four optimized language-model GGUF files for text-only deployment with llama.cpp.
+> **Recommended: M · 1.07 GB** — matches the BF16 strict IFEval reference in non-thinking mode.
+**Qwen 3.5 family:** [0.8B](https://huggingface.co/TheStageAI/Qwen3.5-0.8B-GGUF) · **[2B](https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF)** · [4B](https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF) · [9B](https://huggingface.co/TheStageAI/Qwen3.5-9B-GGUF)
+## Choose a checkpoint
+| Variant | Size | Best for | File |
+| --- | ---: | --- | --- |
+| XS | 738 MB | Minimum footprint · non-thinking only | [GGUF](./Qwen3.5-2B-XS-TS-Q3_K_S.gguf) |
+| S | 967 MB | Compact | [GGUF](./Qwen3.5-2B-S-TS-Q4_K_S.gguf) |
+| **M** | **1.07 GB** | **Recommended · Q4 reference** | [GGUF](./Qwen3.5-2B-M-TS-Q4_K_M.gguf) |
+| L | 2.01 GB | Maximum fidelity | [GGUF](./Qwen3.5-2B-L-TS-Q8_0.gguf) |
+Sizes use decimal MB/GB. Exact byte counts are recorded in [`release-manifest.json`](./release-manifest.json).
 ## Quickstart
+The command below selects the recommended **M** checkpoint by exact filename:
 ```bash
+llama-cli \
+  --hf-repo TheStageAI/Qwen3.5-2B-GGUF \
+  --hf-file Qwen3.5-2B-M-TS-Q4_K_M.gguf
 ```
+## Quality
+| Variant | IFEval prompt / instruction strict (%) | MMLU-Pro (%) |
+| --- | ---: | ---: |
+| BF16 reference | 65.43 / 74.70 | — |
+| XS | 52.68 / 64.15 | — |
+| S | 63.22 / 73.38 | — |
+| **M** | 66.54 / 75.54 | — |
+| L | 65.80 / 74.94 | — |
 - **IFEval:** 541 prompts, native chat template, `enable_thinking=false`, temperature 0.
+- **MMLU-Pro:** complete 12,032-question runs only, native chat template, `enable_thinking=true`, temperature 1, top-p 0.95, 32,768-token output limit.
+- `—` means that no complete headline MMLU-Pro run is reported. Partial-subject accuracies are not used as release scores.
+The benchmarks intentionally exercise different modes: IFEval measures non-thinking instruction and format adherence, while MMLU-Pro includes long sampled reasoning.
+## Reasoning mode
+**XS is not a supported reasoning operating point.** It is released for minimum-footprint, non-thinking chat and instruction following. Use S, M, or L when reasoning quality matters. In Qwen XS, long-thinking evaluation produced unstable trajectories, substantially longer generations, and frequent 32,768-token output-limit hits. Run XS with `--reasoning off`. The full MMLU-Pro matrix for this model was not completed; partial-subject accuracies are not published as release scores.
+## About this release
+This release starts from [`Qwen/Qwen3.5-2B`](https://huggingface.co/Qwen/Qwen3.5-2B) at revision [`15852e8c16360a2fea060d615a32b45270f8a8fc`](https://huggingface.co/Qwen/Qwen3.5-2B/tree/15852e8c16360a2fea060d615a32b45270f8a8fc) and applies TheStageAI's adaptive per-tensor precision assignment with tuned quantization scales.
+- **XS:** minimum-footprint adaptive mixed precision for non-thinking use.
 - **S:** compact adaptive mixed precision.
+- **M:** uniform Q4 tensor assignment and reference operating point.
+- **L:** uniform Q8_0 tensor assignment for maximum fidelity.
+The recommendation is selected separately for each base model from the release evaluations.
+<details>
+<summary><b>Quantization labels and technical details</b></summary>
+| Variant | Hub selector | GGUF file type | Whole-file BPW |
+| --- | --- | --- | ---: |
+| XS | `Q3_K_S` | `MOSTLY_Q2_K` | 3.140 |
+| S | `Q4_K_S` | `MOSTLY_Q2_K` | 4.109 |
+| M | `Q4_K_M` | `MOSTLY_Q4_K_M` | 4.562 |
+| L | `Q8_0` | `MOSTLY_Q8_0` | 8.558 |
+XS and S are TheStage mixed-precision schedules. Their public Q-type suffixes provide an approximate size class for Hub discoverability; exact tensor-type inventories are recorded in [`release-manifest.json`](./release-manifest.json). M and L use uniform Q4_K and Q8_0 assignments for quantized decoder tensors.
+</details>
 ## Limitations
+- These are language-model GGUF files for text-only deployment. Multimodal projector files are not included.
+- File size does not include KV cache or runtime buffers; total memory use grows with context length.
 - Small score reversals between BF16 and quantized rows should be read as evaluation variation, not as a claim that quantization improves the base model.
 ## Provenance
+The exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs are recorded in [`release-manifest.json`](./release-manifest.json). Use its SHA-256 digests to verify downloaded files.
 ## License

release-manifest.json CHANGED Viewed

@@ -28,6 +28,7 @@
     "mmlu_pro": {
       "accuracy": null,
       "correct": null,
       "run_id": null,
       "sample_count": null,
       "status": "not_reported"
@@ -35,12 +36,20 @@
   },
   "display_name": "Qwen3.5 2B",
   "family": "Qwen 3.5",
-  "generated_at": "2026-07-20T21:32:16.184050+00:00",
   "license": "apache-2.0",
   "model_key": "qwen3p5_2b",
   "recommended_product": "M",
   "repo_id": "TheStageAI/Qwen3.5-2B-GGUF",
-  "schema": "thestageai.gguf_release_manifest_v1",
   "variants": [
     {
       "benchmarks": {
@@ -56,6 +65,7 @@
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
@@ -71,8 +81,9 @@
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
-      "positioning": "Minimum size",
       "product": "XS",
       "recommended": false,
       "sha256": "c0d4816bdf93a21f307893675d0c47c083c36fad3bfee49663b91456e8940d30",
       "tensor_count": 320,
@@ -100,6 +111,7 @@
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
@@ -117,6 +129,7 @@
       "parameter_count": 1881825088,
       "positioning": "Compact",
       "product": "S",
       "recommended": false,
       "sha256": "3a7171517f56f500ef8e6fc1d20b56a155cb2e6bd770d0728213298a2d562ea3",
       "tensor_count": 320,
@@ -145,6 +158,7 @@
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
@@ -160,8 +174,9 @@
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
-      "positioning": "Uniform Q4",
       "product": "M",
       "recommended": true,
       "sha256": "8d497863b95e392baf022258f864c34f4a28613df340c500bb647486c52657ae",
       "tensor_count": 320,
@@ -186,6 +201,7 @@
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
@@ -201,8 +217,9 @@
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
-      "positioning": "Uniform Q8",
       "product": "L",
       "recommended": false,
       "sha256": "fe5c7daff8dffebfee0f8d74d108c5ba0d03093eeb41b385916772e55cb27baf",
       "tensor_count": 320,

     "mmlu_pro": {
       "accuracy": null,
       "correct": null,
+      "not_reported_reason": "full_run_not_completed",
       "run_id": null,
       "sample_count": null,
       "status": "not_reported"
   },
   "display_name": "Qwen3.5 2B",
   "family": "Qwen 3.5",
+  "generated_at": "2026-07-20T22:39:04.914041+00:00",
   "license": "apache-2.0",
   "model_key": "qwen3p5_2b",
+  "reasoning_policy": {
+    "reasoning_products": [
+      "S",
+      "M",
+      "L"
+    ],
+    "xs": "non_thinking_only"
+  },
   "recommended_product": "M",
   "repo_id": "TheStageAI/Qwen3.5-2B-GGUF",
+  "schema": "thestageai.gguf_release_manifest_v2",
   "variants": [
     {
       "benchmarks": {
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
+          "not_reported_reason": "unstable_long_thinking_behavior",
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
+      "positioning": "Minimum footprint \u00b7 non-thinking only",
       "product": "XS",
+      "reasoning_support": "non_thinking_only",
       "recommended": false,
       "sha256": "c0d4816bdf93a21f307893675d0c47c083c36fad3bfee49663b91456e8940d30",
       "tensor_count": 320,
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
+          "not_reported_reason": "full_run_not_completed",
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
       "parameter_count": 1881825088,
       "positioning": "Compact",
       "product": "S",
+      "reasoning_support": "supported",
       "recommended": false,
       "sha256": "3a7171517f56f500ef8e6fc1d20b56a155cb2e6bd770d0728213298a2d562ea3",
       "tensor_count": 320,
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
+          "not_reported_reason": "full_run_not_completed",
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
+      "positioning": "Q4 reference",
       "product": "M",
+      "reasoning_support": "supported",
       "recommended": true,
       "sha256": "8d497863b95e392baf022258f864c34f4a28613df340c500bb647486c52657ae",
       "tensor_count": 320,
         "mmlu_pro": {
           "accuracy": null,
           "correct": null,
+          "not_reported_reason": "full_run_not_completed",
           "run_id": null,
           "sample_count": null,
           "status": "not_reported"
       "lm_head_policy": "skip_tied",
       "output_embedding_mode": "tied_alias",
       "parameter_count": 1881825088,
+      "positioning": "Maximum fidelity",
       "product": "L",
+      "reasoning_support": "supported",
       "recommended": false,
       "sha256": "fe5c7daff8dffebfee0f8d74d108c5ba0d03093eeb41b385916772e55cb27baf",
       "tensor_count": 320,