v75 (S24/SM8650, soc 57): Qwen3.5-4B sharded-W8 decode bundle — Plane A greedy-correct, 3.8 tok/s
#2
by flying101 - opened
Sharded W8 decode (s0+s1) + fp16 lmhead/embed, QAIRT 2.47 soc_model 57. Device-validated on S24 (correct greedy answer, 3.8 tok/s).
flying101 changed pull request status to open
flying101 changed pull request title from Add Hexagon v75 (Galaxy S24 / SM8650) bundle to v75 (S24/SM8650, soc 57): Qwen3.5-4B sharded-W8 decode bundle — Plane A greedy-correct, 3.8 tok/s
Adds the Hexagon v75 bundle under .
- Files: qwen3.5-4b-1024.json, qwen354b_decode_s0_w8.bin + qwen354b_decode_s1_w8.bin (2-shard W8, 1.7G each), qwen354b_lmhead_f16.bin, qwen354b_embed_f16.bin, tokenizer.json
- Precision: 2-shard W8 decode + fp16 lm-head/embed (16 query heads -> gqa_native)
- Compile: QAIRT 2.47.0, v75/soc 57; export gate decode 8/8 match HF greedy
- Device (Plane A, S24): each standardized prompt greedy-correct (Paris/Tokyo verified), ~3.8 tok/s; verbose thinking model (needs max_new>=160). The automated 6-in-a-row suite is RAM-limited on the 8GB S24 (per-prompt runs pass).
v75 bundle details: 2-shard W8 decode (1.7G x2) + fp16 lm-head/embed; 16 query heads -> gqa_native. QAIRT 2.47, v75/soc 57, export 8/8. Plane A on S24: each prompt greedy-correct, ~3.8 tok/s (verbose thinker, max_new>=160; 6-in-a-row suite RAM-limited on 8GB).
sanmonga22 changed pull request status to merged