README.md

metadata

license: apache-2.0
tags:
  - qhexrt
  - hexagon
  - npu
  - qnn
  - llm
  - gated-delta-net
base_model: Qwen/Qwen3.5-2B
pipeline_tag: text-generation

Qwen3.5-2B — Hexagon NPU (QHexRT) — v81

A prebuilt QHexRT bundle of Qwen/Qwen3.5-2B (the Qwen3-Next "GatedDeltaNet" hybrid lineage) running on the Qualcomm Hexagon v81 NPU (SM8850). text→text.

The text decoder is a hybrid: 3 of every 4 layers are a recurrent gated delta-rule linear attention (Qwen3_5GatedDeltaNet: a short causal-conv FIFO + an SSM state + gated RMSNorm), and every 4th layer is gated softmax attention (q-proj output gate, per-head q/k RMSNorm, partial-rotary M-RoPE θ=1e7). The runtime drives it with the qwen3_5_generate host-op (decode-over-prompt; conv + SSM + KV state carried host-side), the lm-head as a separate graph.

Two precision bundles — pick the dir

dir	precision	decode bin	decode	parity	when to use
v81/	fp16	2.76 GB	~11.0 tok/s	strict greedy-exact (suite 10/10)	bit-faithful HF parity
v81-w8/	W8A16 (decode W8 + fp16 lm-head)	1.3 GB	~14.4 tok/s	single-prompt greedy-exact 64/64; length-swept suite 8/10 under a tolerant greedy_tol gate	~half the decode size + ~1.3× faster; coherent throughout

Both are device-validated on SM8850 / v81 (QAIRT 2.47). The W8 decode also fits a single context and avoids the fp16 build's >2 GB large-code-model path. Same embed_f16 / lmhead_f16 / tokenizer in both.

What's here

v81/ (fp16 — strict parity)

file	role	~size
qwen3.5-2b-1024.json	QHexRT manifest (the declarative run plan)	2 KB
qwen352b_decode_f16.bin	decode context binary (24 layers, hid 2048, fp16)	2.76 GB
qwen352b_lmhead_f16.bin	lm-head context binary (tied embed, fp16)	1.0 GB
qwen352b_embed_f16.bin	embedding table (host lookup, fp16)	1.0 GB
tokenizer.json	Qwen2-style BPE tokenizer	20 MB

v81-w8/ (W8A16 — smaller + faster)

file	role	~size
qwen3.5-2b-1024.json	QHexRT manifest (references the W8 decode)	2 KB
qwen352b_decode_w8.bin	decode context binary (24 layers, W8 weight-only)	1.3 GB
qwen352b_lmhead_f16.bin	lm-head context binary (tied embed, fp16)	1.0 GB
qwen352b_embed_f16.bin	embedding table (host lookup, fp16)	1.0 GB
tokenizer.json	Qwen2-style BPE tokenizer	20 MB

Arch-pinned: a v81 binary will not load on another Hexagon arch. The QNN runtime libs come from the QAIRT SDK, not this repo.

Run

# fp16 (strict parity):
hf download runanywhere/qwen3_5_2b_HNPU --include "v81/*" --local-dir q35_2b
adb push q35_2b/v81 /data/local/tmp/wq/qwen35-2b
adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
  ./qhx_generate qwen35-2b/qwen3.5-2b-1024.json libQnnHtp.so libQnnSystem.so qwen35-2b 64 'The capital of France is'"

# W8 (smaller + faster):
hf download runanywhere/qwen3_5_2b_HNPU --include "v81-w8/*" --local-dir q35_2b
adb push q35_2b/v81-w8 /data/local/tmp/wq/qwen35-2b-w8
adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
  ./qhx_generate qwen35-2b-w8/qwen3.5-2b-1024.json libQnnHtp.so libQnnSystem.so qwen35-2b-w8 64 'The capital of France is'"

(also stage the QAIRT runtime libs + a qhx_generate built with the qwen3_5_generate host-op — see caveats.)

Measured (device: SM8850 / v81, QAIRT 2.47)

• fp16 (v81/): ~11.0 tok/s; greedy-EXACT vs HF Qwen/Qwen3.5-2B (41/41).
• W8 (v81-w8/): ~14.4 tok/s (~1.3× fp16); single-prompt greedy-exact 64/64; length-swept parity suite (up to 70% of MAXCTX) 8/10 under a tolerant greedy_tol gate (the 2 misses are W8 near-tie argmax-drift into coherent-but-different text — the longer-context variants pass, so it is benign quant noise, not a context bug).

Caveats

• Precision is your choice: v81/ fp16 for strict bit-faithful parity, v81-w8/ for ~half the decode size + ~1.3× speed at a tolerant (8/10 suite) parity. W8 quantizes only weights — the fp32 SSM scan + host conv/SSM/KV state are untouched.
• W4 is HTP-toolchain-blocked on v81 — W8 is the quant floor.
• Text-LLM path only — the vision tower + MTP head are not exported.
• Needs a qhx_generate that includes the qwen3_5_generate host-op (QHexRT branch smonga/qwen_fam).
• Built by the in-repo forge pipeline (oracle-gated export → QAIRT-2.47 O3 compile → device greedy gate).