mando2222/Qwen3-32B-blackhole

Qwen3-32B — Tenstorrent Blackhole (vLLM bundle)

A tt-kernel vLLM bundle that runs Qwen/Qwen3-32B on 4× Tenstorrent Blackhole through the Tenstorrent vLLM plugin. It ships no kernel cache and no weights — only the vLLM registration + launch recipe; weights are pulled from the base repo and kernels JIT-compile at first-run warmup.

Serve

tt-kernel serve mando2222/Qwen3-32B-blackhole

Exposes an OpenAI-compatible server on :8000. Model id Qwen3-32B (aliases qwen3-32b, Qwen/Qwen3-32B). Reasoning parser qwen3 + tool parser hermes enabled; 40960 context; up to 32 concurrent requests.

Exact stack this was built and validated on

Component	Version
tt-metal	branch agentic-research/fast-models-fast + autoport branch autoport/qwen3-32b-bringup @ commit 26296ee8
tt-vllm (fork)	dev branch @ 178f5db + a local plugin patch (see Known limitation)
tt-kernel	PR #4 (vLLM-backend / kernels-less bundles)
vLLM	0.1.dev1+g61bbb1eae (empty-target build; TT provided at runtime)
Hardware	4× Blackhole p300c, opened as a 1×4 mesh (MESH_DEVICE=P300x2)
Driver / firmware	tt-kmd ≥ 2.5, firmware ≥ 19.2; kernel booted with iommu=pt

The precompiled-kernel compatibility key is deliberately omitted (this is a kernels-less bundle), so it is portable across tt-metal builds — kernels recompile at warmup.

Model / precision

Dense Qwen3ForCausalLM, 64 layers, GQA 64/8, head-dim 128, vocab 151936, rope_theta 1e6, 40960 ctx. 1D tensor-parallel (TP=4). Precision: BFP4 + LoFi projection weights, BFP8 paged KV cache, BF16 activations/norms; on-device split-argmax greedy; DRAM-sharded decode matmuls; FABRIC_1D + Ring CCL.

Measured (4× Blackhole)

• Decode: ~26 tok/s/user (single user, 128/128); ~543 tok/s aggregate @ 32 concurrent; TTFT ~91 ms.
• Accuracy: r1_math500 95.0 (pub 96.1); public IFEval strict ~0.79 (≈ 80 ref); vLLM parameter-conformance 21/21 (standalone).

Consumer prerequisites (important)

This bundle is the registration + launch recipe + weights reference, not the full model source. To run it you also need:

1. 4× Blackhole hardware, 1×4 mesh, kernel with iommu=pt (else host↔device DMA degrades).
2. tt-metal built with the autoport code importable at models.autoports.qwen_qwen3_32b.tt.generator_vllm:Qwen3ForCausalLM (it imports models.common.* — a compatible tt-metal is required). This currently lives on the autoport/qwen3-32b-bringup branch, not upstream tt-metal.
3. Tenstorrent vLLM fork (dev) + TT plugin — install via the package manager's scripts/install.sh.

Known limitation

Correct per-request sampling (non-greedy/penalized requests routed to vLLM's host sampler for reproducible seeds + penalties) depends on a local patch to the TT vLLM plugin (vllm_tt_plugin/model_runner.py::check_perform_device_sampling). On a stock plugin, sampled requests are served as on-device greedy (functional, but sampling params under-honored). Greedy decode is unaffected.

License

Weights: Qwen3-32B is Apache-2.0 (see the base repo). This bundle contains only Tenstorrent integration metadata + launch config.