Sparkulator-Laguna-XS-2.1

A W4A16-quantized DFlash speculator for poolside/Laguna-XS-2.1-NVFP4, tuned for memory-bandwidth-bound hardware — specifically a single NVIDIA DGX Spark (GB10, sm_121) serving Laguna-XS 2.1 on one node.

It is a quantization of poolside/Laguna-XS-2.1-DFlash. The weights were not retrained — only their precision changed.

Measured on a DGX Spark: +8.1% decode throughput at a 2K code prompt, with draft acceptance unchanged (65.0% → 65.5%).

Why this exists

Speculative decoding is verified: at temperature 0 a draft token survives only if it equals the target's argmax, otherwise the target's own token is used. So the draft cannot change what the model says — it can only change how fast it says it. That makes the draft a free target for aggressive compression: the only thing at risk is acceptance rate.

On a DGX Spark the binding constraint is memory bandwidth, and the drafter is essentially a weight read repeated k times per verification step. Shrinking it 2.34× shrinks that read.

Target assignment

Use this draft with an NVFP4 Laguna-XS 2.1 target. DFlash drafts ship no lm_head of their own — they share the target's and are calibrated against its hidden states — so a draft is specific to the target it was built for. This one was built from, and measured against, poolside/Laguna-XS-2.1-NVFP4. (Unlike Laguna-S, poolside publishes a single Laguna-XS-2.1-DFlash rather than per-dtype variants.)

How it was made

Data-free group-wise symmetric RTN (group size 128) applied directly to the raw safetensors, emitting a standard compressed-tensors pack-quantized checkpoint that vLLM loads through its marlin W4A16 path. No calibration data, no GPU, no model class.

Quantized	layers.{0..4}.mlp.{gate,up,down}_proj, layers.{0..4}.self_attn.o_proj, fc — 21 tensors
Left BF16	self_attn.qkv_proj, self_attn.g_proj, all norms
Size	0.861 GiB → 0.368 GiB (42.7%)
Mean relative weight error	0.099 – 0.114 per tensor (worst: mlp.down_proj)

qkv_proj stays BF16 deliberately: vLLM's DFlash proposer reads qkv_proj.weight directly when it precomputes the context-KV buffer, with no dequantization path, so quantizing it fails the load with AttributeError: 'QKVParallelLinear' object has no attribute 'weight'.

Measured results

One DGX Spark (GB10, 128 GB unified), vLLM 0.25.1, target poolside/Laguna-XS-2.1-NVFP4, num_speculative_tokens=6, --max-num-seqs 2, KV pinned to 12 GiB, temperature 0, concurrency 1, thinking off, code-continuation workload, warm. Both arms measured back-to-back in the same session with identical flags — only the draft differs.

Prompt	BF16 draft	This draft (W4A16)	Δ
512 tok	131.15 tok/s	136.97 tok/s	+4.4%
2,048 tok	123.52 tok/s	133.55 tok/s	+8.1%
8,192 tok	114.54 tok/s	121.19 tok/s	+5.8%

Acceptance over the measured window (spec_decode_num_accepted_tokens_total / _num_draft_tokens_total from /metrics):

	BF16 draft	This draft
Overall acceptance	65.0%	65.5%
Per position (0→5)	0.90 / 0.80 / 0.70 / 0.60 / 0.49 / 0.43	0.90 / 0.82 / 0.72 / 0.62 / 0.53 / 0.44

Acceptance is not degraded by the quantization — the difference is inside run-to-run noise. Weights loaded: 21.14 GiB (BF16 draft) → 20.64 GiB, i.e. exactly the 0.50 GiB the quantization saves.

Serving it

hf download sapidlabs/Sparkulator-Laguna-XS-2.1 \
  --local-dir ~/models/hf/Sparkulator-Laguna-XS-2.1

VLLM_USE_DEEP_GEMM=0 vllm serve ~/models/hf/Laguna-XS-2.1-NVFP4 \
  --served-model-name laguna-nvfp4 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.5 \
  --kv-cache-memory-bytes 12884901888 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"dflash","model":"'$HOME'/models/hf/Sparkulator-Laguna-XS-2.1","num_speculative_tokens":6}' \
  --host 0.0.0.0 --port 8000

k=6 is the measured peak for this target on a Spark — decode is not monotonic in k.

One patch you need

vLLM 0.25.1's laguna_dflash.py calls self.quant_config.get_cache_scale(name) unconditionally while loading. That method is FP8-only; CompressedTensorsConfig does not have it, so a W4A16 draft dies on load. Guard it (on every node — each has its own site-packages):

# vllm/model_executor/models/laguna_dflash.py
if self.quant_config is not None and hasattr(self.quant_config, "get_cache_scale") and (
    scale_name := self.quant_config.get_cache_scale(name)
):

If you switch between drafts of different quantization without a fresh VLLM_CACHE_ROOT, vLLM reloads a stale compiled graph and dies with KeyError: 'weight_packed' — the compile cache does not re-key on draft quantization.

Credit

All the modelling work is poolside's. poolside/Laguna-XS-2.1-DFlash is their speculator for their model; this repo only changes its numeric precision so it reads fewer bytes per token on a bandwidth-bound box. This is not an official poolside release.

Built as part of How To Spark — see the Laguna-XS 2.1 NVFP4 recipe.