---
license: apache-2.0
base_model: nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: vllm
tags:
  - nvfp4
  - quantized
  - modelopt
  - vllm
  - speculative-decoding
  - mtp
  - dflash
  - blackwell
  - dgx-spark
  - qwen3.6
language:
  - en
---

# Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4

This is an NVFP4 (4-bit floating point) quantization of [nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451), a bf16 `nuslerp` mergekit merge of two Qwen3.6-27B variants, produced with NVIDIA TensorRT Model Optimizer 0.45.0 so it fits and runs on a 128 GB DGX Spark through vLLM's Blackwell FP4 path. No training or fine-tuning was involved: this is a numeric format conversion of somebody else's finished checkpoint. It is still a lossy one, and no quality evaluation of the 4-bit result against the bf16 source was run here (see [Limitations](#limitations)). The two things this card is worth reading for are `mtp.fc.weight` -- the tensor vLLM's multi-token-prediction speculative decoding needs, which is absent from checkpoints pulled before 2026-07-21 and whose absence fails silently -- and the measurement of what MTP and DFlash speculative decoding are actually worth, which turns out to depend far more on the workload than on the concurrency.

---

## Read this first: the missing `mtp.fc.weight`

**If you are using any Qwen3.6-27B checkpoint with MTP speculative decoding, check for `mtp.fc.weight` before you trust a single throughput number.**

The upstream base model was re-uploaded on **2026-07-21T18:30:00Z** (commit `646f37367f`, "Add files using upload-large-folder tool"). That upload added one tensor: `mtp.fc.weight`, shape `[5120, 10240]`, BF16, 104,857,600 bytes: the fusion projection that concatenates the embedding and hidden streams before the MTP head's single decoder layer. The index at an earlier revision (`5ae530c3ab`) has 1198 tensors and no `mtp.fc.weight`; the index at `main` has 1199 and does. Any checkpoint (upstream or a derivative quantization) pulled before that date has `mtp.layers.0.*` and `mtp.norm` but no `mtp.fc`.

**Why this failure is silent.** vLLM only enables its strict weight-initialisation check for *non-quantized* models. From `vllm/model_executor/model_loader/default_loader.py`:

```python
# We only enable strict check for non-quantized models
# that have loaded weights tracking by default.
default_enable_weights_track = (
    model_config.quantization is None and loaded_weights is not None
)
```

On a quantized checkpoint that check is off, so the missing parameter is never reported. `ColumnParallelLinear` allocates it, nothing loads into it, and it keeps whatever the allocation left there. The MTP head then drafts from a random projection: **no exception, no warning, no error in the log.** Acceptance collapses, every drafted token is rejected and re-verified, and you pay the drafting cost for nothing. This card does not contain a timed measurement of that specific broken state, but it does show what very low acceptance costs: the DFlash rows below run at 0.063 acceptance on prose and land 77.0% *below* the no-speculation baseline at N=64. The only symptom is a number you were not measuring.

vLLM's own source treats this tensor as a known special case (`vllm/model_executor/models/qwen3_5_mtp.py`):

```python
# Workaround: mtp.fc is stored as BF16 in NVFP4 checkpoints but is
# missing from hf_quant_config.json exclude_modules. Force unquantized.
# Ref: https://github.com/vllm-project/vllm/pull/38650
# Ref: https://github.com/NVIDIA/Model-Optimizer/pull/1124
```

### Check your own copy

```python
import json
from huggingface_hub import hf_hub_download

repo = "PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4"  # or your own
idx = json.load(open(hf_hub_download(repo, "model.safetensors.index.json")))
wm = idx["weight_map"]

print("tensors:", len(wm))
print("mtp.fc.weight:", "mtp.fc.weight" in wm)
print("mtp.*:", sorted(k for k in wm if k.startswith("mtp.")))
```

`mtp.fc.weight: True` is what you need before enabling MTP. If it prints `False`, do not use `--speculative-config '{"method":"mtp",...}'` with that checkpoint. You will get a silent slowdown, not an error.

### Which build of this model has it

Two artifacts come out of this quantization, and only one of them is safe to run MTP on:

| repo | tensors | `mtp.fc.weight` | MTP |
|---|---:|---|---|
| `PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4` | 2398 | absent | **do not enable** |
| `PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP` | 2399 | present | supported |

Run the snippet above against whichever one you pulled rather than trusting this table. The MTP throughput rows further down were measured on the `-MTP` build; the no-speculation and DFlash rows were measured on the plain `-NVFP4` build, which is immaterial to those two because neither touches the MTP head.

For a local directory, read `model.safetensors.index.json` directly. If you want vLLM to *fail loudly* on any missing tensor rather than trusting the index, force the strict check back on at serve time:

```
--model-loader-extra-config '{"enable_weights_track": true}'
```

(`enable_weights_track` is an accepted key of `--model-loader-extra-config` in vLLM 0.25.x; setting it `true` overrides the quantized-model default and raises `ValueError: Following weights were not initialized from checkpoint: {...}`.)

---

## Serving

All three modes below were run on the same box with the same flags; only `--speculative-config` differs.

Common flags used for every measurement in this card:

```
--kv-cache-dtype fp8
--attention-backend flashinfer
--max-model-len 262144
--gpu-memory-utilization 0.72
--max-num-seqs 64
--reasoning-parser qwen3
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
```

Every measured run also carried `--served-model-name spec-test` and a local request-logging middleware (`--middleware vllm_streams.StreamTracker`) that is not part of vLLM and is not needed to reproduce the serving behaviour; neither is listed above because neither affects how the model is served.

vLLM version under test: **0.25.2.dev0+g752a3a504.d20260721**, in the community Grace-Blackwell image `ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2`.

### 1. No speculative decoding

```bash
vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml
```

The quantization format is read from the checkpoint's own config; no `--quantization` flag is needed.

### 2. MTP (multi-token prediction, in-checkpoint draft head)

Add:

```
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
```

This uses the MTP head already inside the checkpoint (`text_config.mtp_num_hidden_layers: 1`). No second model to download. **Requires a checkpoint carrying `mtp.fc.weight` -- the `-MTP` build, not the plain `-NVFP4` one (see above).**

### 3. DFlash (separate draft model)

Add:

```
--speculative-config '{"method":"dflash","model":"<your-dflash-draft>","num_speculative_tokens":15}'
```

`num_speculative_tokens: 15` is z-lab's own recommended value for this draft.

**The published DFlash draft for this model family does not load on a released vLLM.** [z-lab/Qwen3.6-27B-DFlash](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (1,730,213,120 parameters across 58 tensors, 3,460,432,504 bytes) declares:

```json
"layer_types": ["sliding_attention", "sliding_attention",
                "sliding_attention", "sliding_attention", "full_attention"],
"sliding_window": 2048
```

and `_resolve_layer_attention` in `vllm/model_executor/models/qwen3_dflash.py` raises on any mix of sliding and full:

```
NotImplementedError: DFlash does not yet support mixed sliding/full attention via
layer_types; see https://github.com/vllm-project/vllm/issues/40898.
```

The guard reads the **draft's** `layer_types`, not the target model's, so a stock unmodified Qwen3.6-27B target hits it identically. This is a vLLM gap, not a property of this checkpoint or of any particular target. z-lab's own card acknowledges it and directs users to install vLLM from the branch of that same PR:

```bash
uv pip install -U --torch-backend=auto \
  "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"
```

That is the upstream-sanctioned route, and if you can build from that branch it is the one to use.

**Workaround used for the DFlash numbers below**, for anyone pinned to a released vLLM: rewrite the draft's `config.json` so all five entries read `"sliding_attention"`. The weight file is untouched: the draft used here has SHA-256 `e0c050b34798d32728a164d2c3f1681746ff85c11945701b0205b654e2f1fdbe`, which matches the upstream `model.safetensors` LFS hash exactly. Only `config.json` changed.

That mis-declares one layer: it was trained with full attention and is now given a 2048-token window. The target model verifies every drafted token by rejection sampling, so **output correctness is unaffected either way**; only acceptance can suffer. Note also that a sliding-window layer with a 2048-token window is *identical* to full attention below 2048 tokens of context, so for short contexts the declaration is exact rather than approximate. Every benchmark in this card ran below that threshold (see [Limitations](#limitations)); acceptance beyond 2048 tokens of context is not measured here.

---

## Speculative decoding is not a free speed-up

540 measurement cells: 3 serving configs x 3 workloads x 10 concurrency levels x 6 shuffled repeats, 9,126 streams. All 540 returned status `ok`, with 0 degraded, 0 failed, 0 errored streams. Run window 2026-07-27T20:49:22Z to 2026-07-28T09:39:38Z on one DGX Spark.

The three configs served the same NVFP4 weights, differing only by the single `mtp.fc.weight` tensor described above: the MTP rows come from the `-MTP` build, the base and DFlash rows from the plain `-NVFP4` build. Reproducing the MTP rows requires a checkpoint that has the tensor.

Across the 90 (config, workload, concurrency) groups, the coefficient of variation of aggregate tok/s over the 6 repeats has median 0.42%, p90 2.41%, max 5.20%. Differences below a few percent in the tables below are not meaningful; the large ones are.

### The three workloads

Chosen to differ in one thing: how predictable the next token is. Each pack generates 64 distinct prompts, so no two concurrent streams at N=64 share a prefix (a repeat would hit prefix caching and flatter the result).

| pack | prompt | prompt tokens | intrinsic predictability |
|---|---|---:|---|
| `prose` | "Write a short story of approximately 1000 words about: *&lt;one of 64 topics&gt;*. Tell it in the first person..." | 57-71 | low: every token is a real choice |
| `code` | "Write a single self-contained Python module implementing *&lt;a data structure or algorithm&gt;*... full type hints... at least one runnable doctest" | 79-94 | middle: structured, but generated from scratch |
| `edit` | "Here is a Python module. Make exactly one change: rename every occurrence of `calculate_total` to `compute_total`... every other line must come back byte-for-byte identical" (followed by the module) | 820-849 | near-maximal: the output is almost entirely a copy of the input |

### Acceptance is the variable that explains everything

Acceptance rate is the fraction of drafted tokens the target model accepts, taken as a delta of vLLM's `spec_decode_num_accepted_tokens_total` / `spec_decode_num_draft_tokens_total` counters around each batch.

**Mean acceptance across all ten concurrency levels:**

| method | prose | code | edit |
|---|---:|---:|---:|
| MTP | 0.475 | 0.763 | 1.000 |
| DFlash | 0.063 | 0.187 | 0.827 |

(MTP on `edit` is 0.9998 to four decimal places; it rounds to 1.000 at every level.)

**Acceptance is essentially flat in concurrency.** It moves with *content*, not with load:

| N | MTP prose | MTP code | MTP edit | DFlash prose | DFlash code | DFlash edit |
|---:|---:|---:|---:|---:|---:|---:|
| 1 | 0.492 | 0.723 | 1.000 | 0.066 | 0.160 | 0.830 |
| 2 | 0.467 | 0.783 | 0.999 | 0.060 | 0.199 | 0.828 |
| 4 | 0.491 | 0.785 | 1.000 | 0.065 | 0.203 | 0.833 |
| 6 | 0.461 | 0.771 | 1.000 | 0.062 | 0.199 | 0.827 |
| 8 | 0.476 | 0.787 | 1.000 | 0.061 | 0.202 | 0.833 |
| 12 | 0.469 | 0.786 | 1.000 | 0.064 | 0.204 | 0.824 |
| 16 | 0.473 | 0.752 | 1.000 | 0.063 | 0.184 | 0.830 |
| 24 | 0.473 | 0.747 | 1.000 | 0.063 | 0.170 | 0.823 |
| 32 | 0.470 | 0.751 | 1.000 | 0.064 | 0.171 | 0.822 |
| 64 | 0.475 | 0.748 | 1.000 | 0.064 | 0.175 | 0.820 |

This is the point of the whole exercise, and it is the correction worth making to the usual framing: **speculative decoding is not "a win at low concurrency that decays at high concurrency". It is a win where acceptance is high, and acceptance is a property of how predictable the output is.** Concurrency governs how much *spare compute* is available to absorb rejected drafts, so it sets the size of the penalty, but the sign of the result is set by acceptance. Rewriting a file you were just given (`edit`) accepts almost everything; writing original fiction (`prose`) accepts less than half on MTP and 6% on DFlash. Same box, same flags, same day.

### Aggregate throughput

Aggregate output tokens/second, mean of 6 repeats. `max_tokens=512`, temperature 0.8, thinking disabled; 8,996 of 9,126 streams ran the full budget and 130 stopped early. Token counts come from `usage.completion_tokens`, not SSE chunk counts (under speculation one chunk can carry several tokens, measured up to 13.1 tokens per chunk on DFlash, so chunk-counting under-reports badly).

#### prose

| N | base | MTP | MTP vs base | DFlash | DFlash vs base |
|---:|---:|---:|---:|---:|---:|
| 1 | 12.6 | 16.6 | +32.3% | 16.3 | +29.7% |
| 2 | 25.1 | 32.8 | +30.6% | 26.1 | +4.0% |
| 4 | 48.4 | 61.5 | +27.2% | 43.8 | -9.5% |
| 6 | 70.7 | 84.6 | +19.7% | 53.3 | -24.6% |
| 8 | 91.0 | 108.5 | +19.3% | 61.2 | -32.7% |
| 12 | 127.5 | 112.9 | -11.4% | 72.9 | -42.8% |
| 16 | 160.0 | 144.8 | -9.5% | 78.7 | -50.8% |
| 24 | 212.0 | 228.1 | +7.6% | 70.7 | -66.6% |
| 32 | 251.7 | 262.0 | +4.1% | 77.7 | -69.1% |
| 64 | 353.5 | 328.0 | -7.2% | 81.3 | -77.0% |

#### code

| N | base | MTP | MTP vs base | DFlash | DFlash vs base |
|---:|---:|---:|---:|---:|---:|
| 1 | 12.6 | 20.4 | +62.4% | 27.3 | +117.3% |
| 2 | 25.2 | 42.9 | +70.5% | 49.7 | +97.3% |
| 4 | 48.3 | 80.7 | +66.9% | 79.6 | +64.8% |
| 6 | 69.8 | 111.0 | +59.0% | 98.4 | +41.0% |
| 8 | 92.1 | 144.7 | +57.1% | 115.7 | +25.6% |
| 12 | 127.5 | 149.7 | +17.4% | 138.7 | +8.7% |
| 16 | 159.7 | 175.8 | +10.1% | 136.3 | -14.7% |
| 24 | 209.6 | 282.8 | +34.9% | 131.4 | -37.3% |
| 32 | 249.4 | 326.0 | +30.7% | 140.1 | -43.8% |
| 64 | 351.1 | 403.4 | +14.9% | 145.7 | -58.5% |

#### edit

| N | base | MTP | MTP vs base | DFlash | DFlash vs base |
|---:|---:|---:|---:|---:|---:|
| 1 | 12.4 | 24.7 | +98.7% | 101.5 | +716.1% |
| 2 | 24.7 | 50.3 | +103.9% | 160.7 | +551.0% |
| 4 | 46.9 | 92.6 | +97.7% | 244.3 | +421.3% |
| 6 | 67.7 | 126.7 | +87.1% | 295.6 | +336.5% |
| 8 | 87.5 | 159.8 | +82.8% | 329.8 | +277.1% |
| 12 | 118.9 | 162.3 | +36.5% | 365.1 | +207.0% |
| 16 | 146.1 | 202.7 | +38.8% | 389.8 | +166.9% |
| 24 | 188.0 | 299.5 | +59.3% | 372.8 | +98.3% |
| 32 | 219.2 | 333.7 | +52.3% | 391.9 | +78.8% |
| 64 | 291.8 | 391.5 | +34.2% | 406.6 | +39.4% |

### What the tables say

- **MTP wins at every concurrency level on code and on edit, including N=64**: `code` +14.9% and `edit` +34.2% at N=64, with the largest margins at low concurrency (`code` +62.4%, `edit` +98.7% at N=1). Acceptance on those workloads is 0.763 and 1.000. There is no crossover to a loss.
- **MTP is mixed on prose**, the workload where its acceptance is lowest (0.475). It wins at N=1,2,4,6,8,24,32 and loses at N=12 (-11.4%), N=16 (-9.5%) and N=64 (-7.2%). At half-acceptance the drafting overhead and the win are close enough that the result flips with batch composition.
- **DFlash is enormous on edit-shaped work**: +716.1% single-stream (101.5 vs 12.4 tok/s) and still +39.4% at N=64, winning at every level, on 0.827 acceptance.
- **DFlash hits a hard concurrency ceiling of about 21-22 streams on this box, and its flat lines past that are capacity, not speed.** Reconstructing from the per-stream timings how many generations actually overlapped, base and MTP ran every level they were asked for (achieved/requested = 1.00 at all ten levels); DFlash reached 16 at N=16 and then stuck at 22, 22, 22 for N=24, 32 and 64. Everything above the ceiling **queued**. Confirmed live in the engine log: `Running: 21 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.2%`. So DFlash prose reading 81.3 tok/s at N=64 against base's 353.5 is **not** a decode rate 77% slower - it is roughly 21 streams' worth of throughput while base runs 64. Judged per *running* stream the gap is far smaller; judged as a service it is worse than the number looks, because 43 of the 64 requests are waiting rather than generating. Not a scheduler setting: raising `--max-num-batched-tokens` from the spec-decode default of 2048 to 7296 moved the ceiling by one stream (22 to 21). It is KV capacity - see the next section.
- **Because of that ceiling, DFlash's aggregate curve saturates on every workload**, including the ones it wins: edit goes 389.8 tok/s at N=16 to 406.6 at N=64 (+4%) while base goes 146.1 to 291.8 (+100%). DFlash still wins edit at every level, but it wins by a shrinking margin for a structural reason rather than a workload one. On code it wins N=1 through N=12 and trails from N=16 up, which is exactly where the ceiling starts to bind.
- **The governing variable is acceptance, not concurrency.** The acceptance table above is flat across N; the outcome is not. Rank the three workloads by acceptance and you have ranked them by result, for both methods, at every level.

### The KV cost nobody mentions

Speculative decoding takes memory out of the KV pool, which directly limits how much context you can serve. Read from the engine at load time (structural, not timed), at `--max-model-len 262144` and `--gpu-memory-utilization 0.72` on a 121.7 GiB unified-memory box:

| config | weights loaded | KV cache tokens | KV cache | max concurrency at 262144 ctx |
|---|---:|---:|---:|---:|
| base | 18.76 GiB | 2,041,963 | 63.75 GiB | 7.79x |
| MTP | 19.55 GiB | 1,839,553 | 63.02 GiB | 7.02x  (-10%) |
| DFlash | 22.16 GiB | 1,233,978 | 59.89 GiB | 4.71x  (-40%) |

MTP costs about 10% of the KV pool. DFlash costs about 40%, because it loads a second 3.46 GB model *and* needs its own KV. If your deployment is context-bound rather than throughput-bound, that is the number to budget against, and it is fixed regardless of workload.

**This is also the cause of DFlash's ~21-stream ceiling**, and the two costs compound. The pool is 40% smaller, and each running sequence additionally reserves lookahead slots for the 15 draft tokens it proposes per step, so the per-stream KV footprint roughly doubles: dividing the pool by the streams that actually fit gives about 31,900 tokens per stream under base and about 58,800 under DFlash. Fewer tokens shared among hungrier sequences is why the engine reports `GPU KV cache usage: 97.2%` at 21 concurrent requests and queues the rest. Budget for it directly: on this hardware, at these settings, DFlash serves roughly a third as many simultaneous streams as no-speculation does, whatever the workload. Both figures scale with `--gpu-memory-utilization` and shrink further at longer `--max-model-len`.

### Per-stream fairness and TTFT at N=64

Aggregate throughput says what the box delivers; it says nothing about what one user waiting on one stream experiences. Per-stream tok/s at N=64 (min / p05 / p50 / p95 / max, mean of 6 repeats):

| workload | base | MTP | DFlash |
|---|---|---|---|
| prose | 5.57 / 5.60 / 5.64 / 5.66 / 5.66 | 5.25 / 5.28 / 5.44 / 5.67 / 5.84 | 3.46 / 3.57 / 3.90 / 4.33 / 4.46 |
| code | 5.55 / 5.58 / 5.64 / 5.66 / 5.66 | 6.54 / 6.73 / 6.99 / 7.38 / 7.54 | 5.08 / 5.60 / 7.34 / 10.50 / 12.28 |
| edit | 4.75 / 4.79 / 5.10 / 5.49 / 5.52 | 6.71 / 6.79 / 7.42 / 8.52 / 8.60 | 19.62 / 20.20 / 21.77 / 30.72 / 33.78 |

Base is almost perfectly uniform (max/min 1.02 on prose). Speculation makes streams *unequal*, because acceptance varies per stream: DFlash on code spans 5.08 to 12.28 tok/s in the same batch.

Time to first token is the clearest symptom of the ceiling, and it is worth reading as a *separate* cost from decode speed: waiting to start and generating slowly are different failures. Median TTFT at N=64: prose 1.9 s base / 2.6 s MTP / **138.1 s DFlash**; code 2.3 / 3.1 / **74.9 s**; edit 10.6 / 12.5 / 31.9 s. Base and MTP admit all 64 requests immediately, so their TTFT stays flat; DFlash's median request waits over two minutes for a slot, which no throughput figure shows.

Note this also explains why the DFlash per-stream numbers above look healthier than its aggregate: per-stream rate is measured over each stream's own decode window and therefore **excludes queue time entirely**. A DFlash stream is fast once it starts. Most of them just start late. If you are serving interactive traffic, size your concurrency to stay under the ceiling rather than relying on the queue.

---

## Provenance and reproduction

**Source.** [nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451): bf16, 1199 tensors, 55,457,998,304 bytes declared in its index. Per its own `mergekit_config.yml` it is a `nuslerp` merge of `Qwen3.6-27B-Architect-Polaris2-Fable-B` (weight 1.4) and `Qwen3.6-27B-Architect-Polaris-Fable-F451` (weight 0.6), `dtype: bfloat16`. Uploaded with nightmedia's permission.

**Architecture.** `Qwen3_5ForConditionalGeneration`, `model_type: qwen3_5`. Hybrid attention: 64 decoder layers, 48 `linear_attention` (gated-delta / Mamba-style) and 16 `full_attention`, one full-attention layer every fourth (`full_attention_interval: 4`, at indices 3, 7, ... 63). Hidden size 5120, 24 attention heads, 4 KV heads, head dim 256, intermediate 17408. Vocabulary 248320, `max_position_embeddings` 262144. Vision tower present (`language_model_only: false`), so this is a vision-language checkpoint; MTP head is one decoder layer (`mtp_num_hidden_layers: 1`).

**Quantization.** NVIDIA TensorRT Model Optimizer **0.45.0** (`producer: {"name": "modelopt", "version": "0.45.0"}` in `hf_quant_config.json`), calling `examples/llm_ptq/hf_ptq.py` directly rather than the `huggingface_example.sh` wrapper, which never forwards a dataset flag and falls back to a gated default:

```bash
python hf_ptq.py \
  --pyt_ckpt_path='<bf16 checkpoint>' \
  --export_path='<out>' \
  --qformat=nvfp4 \
  --calib_size=512 \
  --batch_size=0 \
  --inference_tensor_parallel=1 \
  --dataset cnn_dailymail
```

Calibration: **cnn_dailymail** (public), **512 samples**. Container: `nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev` on the DGX Spark itself.

`quant_algo: NVFP4`, `kv_cache_quant_algo: FP8`, `group_size: 16`, weights and input activations both 4-bit float. ModelOpt selected 149 exclude patterns automatically: `lm_head`, `model.language_model.embed_tokens`, all 48 `linear_attn.conv1d` / `in_proj_a` / `in_proj_b` blocks, `model.visual*`, and `mtp*` / `mtp.layers.0*`. The linear-attention, vision and MTP paths stay at higher precision; the full-attention and MLP linears are the ones that went to NVFP4.

**Sizes.** Input 1199 tensors / 55,457,998,304 bytes bf16. ModelOpt export 2398 tensors / 20,487,991,904 bytes -- this is the plain `-NVFP4` repo. The `-MTP` build is that same export plus `mtp.fc.weight` (104,857,600 bytes, carried through unquantized as BF16 exactly as vLLM expects), giving **2399 tensors / 20,592,849,504 bytes**. The two differ by that one tensor and nothing else.

**Benchmark harness.** `specbench.py` (matrix runner), `workloads.py` (the three prompt packs), `report.py` / `dashboard.py` (charts), plus the raw 540-cell `overnight.json` with every per-stream record behind these tables:

> **TODO before publishing: replace with the harness repository URL.** `https://github.com/PLACEHOLDER/PLACEHOLDER`

The harness is host-side, stdlib-only Python 3.12. It drives `docker` directly; records `usage.completion_tokens` rather than SSE chunks; flags any cell with a missing usage block or a generation that stopped short as `degraded` rather than reporting it as a clean number; and uses at least 64 distinct prompts per pack so no two concurrent streams share a prefix (a repeat would hit prefix caching and flatter the result). Reproducing the exact matrix:

```bash
python3 specbench.py --workloads prose,code,edit \
  --levels 1,2,4,6,8,12,16,24,32,64 --repeats 6 --out overnight.json
```

---

## Credits

- **[nightmedia](https://huggingface.co/nightmedia)**: the base merge, [Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451), and its Architect-Polaris merge stages. All the model work is theirs. Uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B, DavidAU's Qwen3.5/3.6-27B finetunes, and armand0e's Qwen3.6-27B-Fable-5-Experimental.
- **[Qwen](https://huggingface.co/Qwen)** (Alibaba): [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B), the foundation everything above is built on. Apache-2.0.
- **[z-lab](https://huggingface.co/z-lab)**: [Qwen3.6-27B-DFlash](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash), the DFlash draft model measured here (MIT licence, per its own repo metadata). The weights used were byte-identical to theirs; only `config.json` was rewritten to work around a vLLM loader gap.
- **NVIDIA**: [TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.45.0, which did the quantization, and the NVFP4 format.
- **vLLM** and the community Grace-Blackwell image `ghcr.io/timothystewart6/vllm-gb10`.

**Licence:** Apache-2.0, inherited from the base model, which declares `license: apache-2.0`, as does Qwen/Qwen3.6-27B. The DFlash draft is separately MIT-licensed by z-lab and is not redistributed here.

---

## Limitations

Read these before generalising any number above.

- **One box, one run.** Every measurement comes from a single DGX Spark: GB10, sm_121a, arm64, **121.7 GiB unified memory shared between CPU and GPU**. A discrete-GPU system with separate HBM will not reproduce the KV-pool arithmetic, and probably not the crossover points either. There was no second machine and no second run of the full matrix.
- **One model.** These are acceptance rates for *this merge*, whose MTP head was inherited through a mergekit merge and **never retrained against the merged weights**. Another Qwen3.6-27B derivative, or the stock Qwen release, will have a different MTP acceptance, possibly much better. Do not read 0.475 on prose as a property of MTP.
- **Thinking was disabled.** Every request passed `chat_template_kwargs: {"enable_thinking": false}`. Reasoning-mode output has different token statistics and would plausibly have different acceptance.
- **Short sequences only.** Prompts are 57-71 tokens (prose), 79-94 (code) and 820-849 (edit); generation was capped at 512 tokens. The longest sequence in the entire matrix is under 1,400 tokens. Nothing here measures behaviour at 32k, 128k or 262k context, and in particular the DFlash draft's 2048-token sliding window was never exceeded, so the `layer_types` rewrite described above was exact for these runs and is untested beyond 2048 tokens of context.
- **Community image, unknown local patches.** vLLM `0.25.2.dev0+g752a3a504.d20260721` inside `ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2`. This is a third-party Grace-Blackwell build, not an official vLLM release; it may carry patches that affect these numbers. A stock vLLM on other hardware may differ.
- **`num_speculative_tokens` was not swept.** MTP ran at 2, DFlash at 15. Both are plausible defaults, neither was tuned per workload, and the crossover points would move if they were.
- **Throughput only.** The workload graders check that responses are complete and non-refusing; they do not score quality. This card contains no evidence about whether NVFP4 degrades output quality relative to the bf16 source, and no benchmark scores of any kind.
- **Text-only testing.** The vision tower is present and was left unquantized, and `pipeline_tag` is inherited from the base model, but every request in this card was text. Image and video input were not exercised at all.
- **Format conversion, not a retrain.** Whatever the original merge does well or badly, it still does. Its behaviour, style, biases and failure modes carry through unchanged.