README.md

metadata

license: mit
tags:
  - qhexrt
  - hexagon
  - npu
  - qnn
  - llm
  - deepseek-r1
  - qwen2
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
pipeline_tag: text-generation

DeepSeek-R1-Distill-Qwen-1.5B — Hexagon NPU (QHexRT) bundle

A prebuilt QHexRT bundle that runs DeepSeek-R1-Distill-Qwen-1.5B fully on the Qualcomm Hexagon NPU (no CPU/GPU in the compute loop). Drop the arch dir on a device and run with qhx_generate.

• Arch present: v81/ — Hexagon v81 (Snapdragon 8 Elite Gen 2 class, SM8850, soc_model 87).
• Arch-pinned: a context binary won't load on another Hexagon arch (the soc_model+dsp_arch are baked in). Each arch is its own flat <arch>/ dir.

What's optimized

• GQA-native decode (12 query / 2 KV heads grouped, no repeat_interleave), in-graph attention with a fixed-MAXCTX (1024) KV cache + 1D causal mask, NPU lm-head, host embed lookup + RoPE (θ=1e4) + sampler.
• fp16 throughout (decode/prefill/lm-head). W8 weight-only quant is available but degrades this small model (first token correct, continuation drifts) — fp16 is the validated greedy-exact bundle.
• v81 f16-attention-overflow fix (critical for this family): Qwen2/2.5 has no QK-norm, so q/k stay unnormalized (|k| up to ~400) and the attention q·k products overflow fp16 (>65504) on the HTP → garbage. The export folds the 1/sqrt(d) scale into q before the matmul and computes the self-score as a matmul (HMX f32-accumulate, not an fp16 reduce). Math-identical; makes decode greedy-exact on v81.

Files (v81/)

file	role
DeepSeek-R1-Distill-Qwen-1.5B.json	QHexRT manifest (the declarative run plan)
deepseekr1distillqwen15b_prefill_f16.bin	prefill context (fp16)
deepseekr1distillqwen15b_decode_f16.bin	decode context (fp16)
deepseekr1distillqwen15b_lmhead_f16.bin	lm-head context (fp16)
deepseekr1distillqwen15b_embed_f16.bin	input embedding table (fp16, host lookup)
tokenizer.json	tokenizer

Run

hf download runanywhere/deepseek_r1_distill_qwen_1_5b_HNPU --local-dir ds15b
adb push ds15b/v81 /data/local/tmp/wq/ds15b           # PowerShell + native paths on Windows
# QNN runtime libs (libQnnHtp.so/libQnnSystem.so + v81 HTP skel) come from the QAIRT SDK, not this repo
adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
  ./qhx_generate ds15b/DeepSeek-R1-Distill-Qwen-1.5B.json libQnnHtp.so libQnnSystem.so ds15b \
  64 'The capital of France is'"

Tool arg order is invariant: <tool> <manifest> libQnnHtp.so libQnnSystem.so <artifacts_root> ….

Measured (v81 / SM8850, qhx_generate)

• Parity: greedy-exact vs the HF fp32 reference — 10/10 gold tokens ("The capital of France is" → " Paris, and the capital of Germany is Berlin."), and generalizes (e.g. "What is 15 multiplied by 6?" → coherent step-by-step reasoning).
• Decode: ~13.5 tok/s (73.9 ms/tok), MAXCTX 1024.

Caveats

• fp16 bundle (validated). W8 weight-only was tried but degrades this small model (W4 is HTP-toolchain-blocked); fp16 is shipped for correctness.
• Decode-over-prompt seeding (no batched-prefill path at runtime). v81-pinned — won't load on other Hexagon arches; re-finalize per arch.