Instructions to use klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP")
model = AutoModelForMultimodalLM.from_pretrained("klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP

SGLang

How to use klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP with Docker Model Runner:
```
docker model run hf.co/klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP
```

Qwen3.8-Flash-Next AutoRound A100 3 bpw + MTP

This is a public, mixed-precision AutoRound checkpoint of Qwen/Qwen3.8-Flash-Next, pinned to revision de4b8e4d43b917e7706784d8bb445c9af86a3540. It targets one NVIDIA A100 (SM80) with a 64 GiB GPU-resident weight budget.

Qwen3.8-Flash-Next is an experimental preview model. Runtime support is also experimental; read the compatibility section before downloading.

Quantization

48 routed-expert banks: depth-stratified 2.998 effective bpw (scale/zero-point overhead included): W3A16G128 on layers 0-11 and 36-47, W2A16G64 on layers 12-35.
Backbone QSA/GDN major linear projections: W8A16G128.
Routers, shared experts, hyperconnection/control paths, PLE projections, vision tower, embeddings, and LM head: BF16.
MTP experts: W4A16G128 symmetric RTN; MTP QSA and dense projections: W8A16G128 symmetric RTN.
Backbone packing: symmetric RTN through AutoRound's shard-streamed model-free path (no calibration dataset).
Packing: native AutoRound auto_round:auto_gptq mixed-bit format.

The module-level precision metadata contains: {"16": 858, "3": 24, "4": 1536, "8": 175}.

Measured storage topology

GPU-resident weight tensors: 48.52 GiB.
Host-offloaded PLE n-gram table: 95.37 GiB (128 tensors).
MTP tensors are included in the checkpoint and counted in the GPU-resident figure.
Safetensor shards: 133; indexed tensors: 227702.

The 64 GiB figure is a weight budget, not a claim that every context length fits. KV cache and runtime workspaces require additional HBM. The 95 GiB n-gram table must remain in host memory; provision ample system RAM.

Runtime compatibility

At publication time, stock vLLM does not yet contain both required changes. Use a build combining:

Serve with tensor parallel size 1. Keep PLE CPU offload enabled and MTP GPU-resident. Set the runtime's GPU memory limit according to the desired KV-cache headroom; do not assume that a 64 GiB weight fit implies a 64 GiB total process fit.

Validation

The official source checkpoint passed a complete 131-shard/index inventory check before quantization. This repository passed a second safetensor/index scan, mixed-bit metadata checks, MTP presence checks, source-code exclusion, and the measured 64 GiB resident-weight gate. See source_validation.json, validation_report.json, build_info.json, and mtp_pack_plan.json for machine-readable details.

Quantization changes model outputs and may reduce quality, especially on workloads sensitive to 2-bit expert banks. No accuracy benchmark is claimed here.

License

Apache-2.0, inherited from the base model. Follow the base model card and license for acceptable use and limitations.

Downloads last month: -

Safetensors

Model size

66B params

Tensor type

BF16

I32

F16

I64

Model tree for klee100/Qwen3.8-Flash-Next-AutoRound-A100-3bpw-MTP

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(85)

this model