Upload README.md with huggingface_hub

README.md

@@ -16,7 +16,19 @@ The text decoder is a **hybrid**: 3 of every 4 layers are a **recurrent gated de
 runtime drives it with the `qwen3_5_generate` host-op (decode-over-prompt; conv + SSM + KV state carried
 host-side), the lm-head as a separate graph.
 
-## What's here (`v81/` — the flat artifacts root)
+## Two precision bundles — pick the dir
+
+| dir | precision | decode bin | decode | parity | when to use |
+|---|---|---|---|---|---|
+| **`v81/`** | **fp16** | 2.76 GB | ~11.0 tok/s | **strict greedy-exact** (suite 10/10) | bit-faithful HF parity |
+| **`v81-w8/`** | **W8A16** (decode W8 + fp16 lm-head) | **1.3 GB** | **~14.4 tok/s** | single-prompt greedy-exact 64/64; **length-swept suite 8/10** under a tolerant `greedy_tol` gate | **~half the decode size + ~1.3× faster**; coherent throughout |
+
+Both are device-validated on **SM8850 / v81** (QAIRT 2.47). The W8 decode also **fits a single context** and
+avoids the fp16 build's >2 GB large-code-model path. Same `embed_f16` / `lmhead_f16` / `tokenizer` in both.
+
+## What's here
+
+### `v81/` (fp16 — strict parity)
 | file | role | ~size |
 |---|---|---|
 | `qwen3.5-2b-1024.json` | QHexRT manifest (the declarative run plan) | 2 KB |
@@ -25,27 +37,46 @@ host-side), the lm-head as a separate graph.
 | `qwen352b_embed_f16.bin` | embedding table (host lookup, fp16) | 1.0 GB |
 | `tokenizer.json` | Qwen2-style BPE tokenizer | 20 MB |
 
+### `v81-w8/` (W8A16 — smaller + faster)
+| file | role | ~size |
+|---|---|---|
+| `qwen3.5-2b-1024.json` | QHexRT manifest (references the W8 decode) | 2 KB |
+| `qwen352b_decode_w8.bin` | decode context binary (24 layers, **W8 weight-only**) | **1.3 GB** |
+| `qwen352b_lmhead_f16.bin` | lm-head context binary (tied embed, fp16) | 1.0 GB |
+| `qwen352b_embed_f16.bin` | embedding table (host lookup, fp16) | 1.0 GB |
+| `tokenizer.json` | Qwen2-style BPE tokenizer | 20 MB |
+
 Arch-pinned: a v81 binary will not load on another Hexagon arch. The QNN runtime libs come from the QAIRT SDK,
 not this repo.
 
 ## Run
 ```bash
-hf download runanywhere/qwen3_5_2b_HNPU --local-dir q35_2b
+# fp16 (strict parity):
+hf download runanywhere/qwen3_5_2b_HNPU --include "v81/*" --local-dir q35_2b
 adb push q35_2b/v81 /data/local/tmp/wq/qwen35-2b
-# also stage the QAIRT runtime libs + a qhx_generate built with the qwen3_5_generate host-op (see caveats)
 adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
   ./qhx_generate qwen35-2b/qwen3.5-2b-1024.json libQnnHtp.so libQnnSystem.so qwen35-2b 64 'The capital of France is'"
+
+# W8 (smaller + faster):
+hf download runanywhere/qwen3_5_2b_HNPU --include "v81-w8/*" --local-dir q35_2b
+adb push q35_2b/v81-w8 /data/local/tmp/wq/qwen35-2b-w8
+adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
+  ./qhx_generate qwen35-2b-w8/qwen3.5-2b-1024.json libQnnHtp.so libQnnSystem.so qwen35-2b-w8 64 'The capital of France is'"
 ```
+(also stage the QAIRT runtime libs + a `qhx_generate` built with the `qwen3_5_generate` host-op — see caveats.)
 
 ## Measured (device: SM8850 / v81, QAIRT 2.47)
-- **Decode: ~11.0 tok/s** (fp16).
-- **Parity: greedy-EXACT** — on-device tokens match the HF `Qwen/Qwen3.5-2B` greedy continuation **41/41**
-  (prompt "The capital of France is").
+- **fp16 (`v81/`):** ~11.0 tok/s; greedy-EXACT vs HF `Qwen/Qwen3.5-2B` (41/41).
+- **W8 (`v81-w8/`):** **~14.4 tok/s** (~1.3× fp16); single-prompt greedy-exact **64/64**; length-swept parity
+  suite (up to 70% of MAXCTX) **8/10** under a tolerant `greedy_tol` gate (the 2 misses are W8 near-tie
+  argmax-drift into coherent-but-different text — the longer-context variants pass, so it is benign quant
+  noise, not a context bug).
 
 ## Caveats
-- **fp16** decode + lm-head (strict greedy parity). W8 weight-only is the next perf lever (~2×, weight-bound).
-- The fp16 decode weights (2.7 GB) exceed the 2 GB model-lib relocation limit, so the build uses a
-  **large-code-model** path (weight blob → `.lrodata` + `-mcmodel=large`); this is baked into the published bin.
+- **Precision is your choice:** `v81/` fp16 for strict bit-faithful parity, `v81-w8/` for ~half the decode
+  size + ~1.3× speed at a *tolerant* (8/10 suite) parity. W8 quantizes only weights — the fp32 SSM scan +
+  host conv/SSM/KV state are untouched.
+- **W4 is HTP-toolchain-blocked** on v81 — W8 is the quant floor.
 - **Text-LLM path only** — the vision tower + MTP head are not exported.
 - Needs a `qhx_generate` that includes the **`qwen3_5_generate`** host-op (QHexRT branch `smonga/qwen_fam`).
-- Built by the in-repo `forge` pipeline (oracle-gated export 10/10 → QAIRT-2.47 O3 compile → device greedy gate).
+- Built by the in-repo `forge` pipeline (oracle-gated export → QAIRT-2.47 O3 compile → device greedy gate).