Instructions to use gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8")
model = AutoModelForMultimodalLM.from_pretrained("gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8

SGLang

How to use gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8 with Docker Model Runner:
```
docker model run hf.co/gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8
```

Qwen3.8-Flash-Next-Abliterated-NVFP4-plefp8

windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 with its 51.2B PLE n-gram embedding table stored in FP8 instead of BF16, and its PLE tensors re-laid-out into the shard form vLLM's loader expects.

173.6 GiB → 125.9 GiB. Nothing else is touched: the NVFP4 body, the abliteration, the vision tower and the MTP draft head are all copied through byte-for-byte.

On 2× DGX Spark (TP2) that is 88.63 → 64.79 GiB per node, and the freed memory goes almost entirely to KV cache.

Why the table dominates the file size

Qwen3.8-Flash-Next is 125B of body plus a 51.2B N-gram (PLE) embedding table living in a single layer. The table is a hashed lookup, not a matmul, so every NVFP4 quantizer excludes it from the 4-bit pass. What differs between published NVFP4 builds is therefore mostly what dtype the excluded table is stored in:

checkpoint	PLE table	body	total
Qwen/Qwen3.8-Flash-Next-FP8	FP8	FP8	172.8 GiB
Inferact/Qwen3.8-Flash-Next-NVFP4	BF16	NVFP4	170.2 GiB
RadixArk/Qwen3.8-Flash-Next-NVFP4	FP8	NVFP4	125.9 GiB
upstream of this repo	BF16	NVFP4	173.6 GiB
this repo	FP8	NVFP4	125.9 GiB

NVFP4 shrinks the experts in every 4-bit build; a BF16 table hands the gain back.

What changed, exactly

1. PLE table split and renamed. Upstream ships the table as one consolidated tensor:

model.language_model.layers.1.ple.ple_embedding.shard_.weight   BF16 [320001536, 160]  95.37 GiB

vLLM's ple_layer.load_weights only recognises ngram_embedding.shard_<int>.weight. That name has neither the ngram_embedding. component nor a shard index, so it falls through to AutoWeightsLoader, which has no parameter to bind it to. Here it is split into split_ngram_parts = 128 shards of 2,500,012 rows (320,001,536 / 128) named the way the loader expects — the same shape and naming Qwen's own FP8 build uses.

2. Cast BF16 → float8_e4m3fn with a single global per-tensor scale, written as ...ngram_embedding.weight_scale, bf16, shape (1,) — again matching Qwen's FP8 build. The computed scale is 0.00019931793213, which is bit-identical to the scale in Qwen's own FP8 checkpoint (abliteration barely moves the table's dynamic range). Round-trip vs the source table: cosine 0.9997, max relative error ~3.5%.

3. layer_types normalised. Upstream's config (emitted by transformers 5.16.0.dev0) labels the 12 QSA layers qwen_sparse_attention. vLLM rejects that outright:

ValidationError: `layer_types` entries must be in (...) but got [... 'qwen_sparse_attention' ...]

The 12 layer positions are identical to Qwen's published config ([3, 7, 11 … 47]), so they are renamed to full_attention. This fix is needed to serve the upstream checkpoint too — it is not specific to the FP8 repack.

Does it need a patched vLLM?

Yes — one small patch. vLLM selects its FP8 PLE embedding method only when the global quantization config is Fp8Config:

def _get_ple_embedding_quant_method(quant_config, prefix):
    if not isinstance(quant_config, Fp8Config):
        return None

This checkpoint's global config is ModelOptNvFp4Config (the body is NVFP4) with *.ple.* in exclude_modules, so that returns None, the embedding is built in BF16, and loading the F8_E4M3 shards fails. The patch decides from the on-disk tensor dtype instead, which leaves BF16-table checkpoints on the unquantized path unchanged.

The patch is included as patches/fix-qwen38fn-nvfp4-ple-fp8.sh (idempotent, self-verifying) and is also carried in eugr/spark-vllm-docker as mods/fix-qwen38fn-nvfp4-ple-fp8.

Everything else — the NVFP4 body, vision tower, MTP head — loads on a stock qwen4_exp-capable vLLM.

Measured: what the FP8 table costs

Both builds are identical except for the table's dtype (same weights, same shard layout, same config fix), served on 2× DGX Spark, TP2, 262144 context:

	BF16 table (control)	FP8 table (this repo)
checkpoint	173.59 GiB	125.90 GiB
weights / node	88.63 GiB	64.79 GiB
KV cache	9.47 GiB	33.20 GiB
KV tokens	636,230	2,199,175
max concurrency @262k	2.43×	8.39×
decode (single stream)	44.3 / 51.7 / 58.0 tok/s	42.8 / 53.9 / 54.9 tok/s
MoE backend	MARLIN	MARLIN

Quality, first-token distribution over 80 harmless prompts, BF16 build as reference:

metric	value
top-1 token agreement	100.0%
mean KL	0.0018
median KL	0.0006
p90 KL	0.0060

Long-generation degeneracy (20 runs each) and tool-calling (24 cases each) show no regression: mean_distinct_3 0.9888 → 0.9894, max_ngram_run 1 → 1, json_parse_rate 1.00 → 1.00.

The FP8 build picks the same first token as the BF16 build on every prompt tested. For context, the NVFP4 body quantization both builds already carry is roughly an order of magnitude lossier than this cast.

Serving (vLLM, 2× DGX Spark)

vllm serve <path-to-this-repo> \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.86 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL"}' \
  --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Notes for GB10 / DGX Spark specifically:

mode:0 disables Inductor. With compilation on, its autotuner tries to benchmark a kernel over the whole PLE table and asks for ~47.7 GiB in one allocation, which OOMs. cudagraph_mode: FULL still gives graph replay.
VLLM_PLE_CPU_OFFLOAD=1 is single-node only (Unsupported settings: nnodes=2), so it cannot be used on a 2-node cluster. On unified memory it would not help with capacity anyway — host and device are the same physical pool.
gpu_memory_utilization ceiling is lower than it looks: CUDA's free memory tracks the OS free column, not available, so page cache eats into the budget.

Reproducing

python3 repack_ple_fp8.py \
  --src <windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 snapshot> \
  --dst <output dir>
# control build (split + rename, keep BF16):
python3 repack_ple_fp8.py --src <snapshot> --dst <dir> --keep-bf16

repack_ple_fp8.py is included here and in eugr/spark-vllm-docker under tools/ple-repack/. It streams the table in row slices, hardlinks every untouched shard file, and rewrites the index — it never holds the 95 GiB table in memory.

Credits

Qwen — Qwen3.8-Flash-Next, the base model, and the FP8 build whose PLE layout and scale convention this reproduces.
windowsxp811203 — the abliteration and the NVFP4 conversion this repo is derived from. Their ARCHITECTURE.md (included) is the best documentation of qwen4_exp internals I have found, and their graft_nvfp4.py / fix_nvfp4_config.py document real failure modes in quantizing this architecture (silently dropped vision and MTP tensors). Both are intact here: 333 vision tensors and 31 MTP tensors verified present.
RadixArk — their NVFP4 build demonstrated that an FP8 PLE table is viable, which is what prompted this repack.

Only the PLE table's dtype and layout differ from the upstream abliterated NVFP4 build. All credit for the model itself belongs upstream; any mistakes in the repack are mine.