---
license: apache-2.0
tags:
  - qhexrt
  - qwen3
  - hexagon
  - npu
  - qnn
  - ternary
base_model: prism-ml/Ternary-Bonsai-1.7B-unpacked
pipeline_tag: text-generation
language:
  - en
---

# Ternary-Bonsai-1.7B — QHexRT NPU bundle (Hexagon v79 + v81)

Precompiled **Ternary-Bonsai-1.7B** (prism-ml's ternary-**trained**, dense **Qwen3-1.7B**) for the **QHexRT**
runtime on Qualcomm Hexagon. Two arch-pinned bundles in this repo:

| arch | dir | device | status |
|------|-----|--------|--------|
| **v81** | `v81/` | SM8850 / Snapdragon 8 Elite (e.g. Galaxy S26) | **device-validated** — 64/64 greedy-exact vs HF @ ~18.7 tok/s |
| **v79** | `v79/` | SM8750 / Snapdragon 8 Elite (e.g. Galaxy S25) | **compiled** (same oracle-gated path, export gate 8/8); on-device gate **pending a v79 device** |

Runs **entirely on the NPU**: the 28-layer transformer **and** the tied lm-head execute on HTP/HMX via the
`qwen3_generate` host-op (GQA-native, resident-KV decode); the host does only embed lookup, **YaRN RoPE**, the
causal mask, and sampling.

> **Ternary → W8, not 1-bit.** The published ternary {−1,0,+1} packing is below the Hexagon quant floor
> (W8A16; W4 is toolchain-blocked, sub-byte HMX impossible). This bundle runs prism's **ternary-trained
> weights at W8** (the `-unpacked` dense form) — it keeps the trained accuracy and is NPU-resident, but is
> **not** 1-bit on-device.

## What's optimized
- **GQA-native W8 decode** (16 Q / 8 KV heads, no KV repeat → no VTCM spill) + an **in-graph W8 tied lm-head**.
- **YaRN RoPE** baked into the manifest (`rope_inv_freq` + `rope_mscale`) and applied **host-side on the ARM
  CPU**, so it is arch-identical across v79/v81 (the two manifests differ only in `dsp_arch`).
- W8A16 weight-only @ O3, VTCM 8 MB; decode-over-prompt seed (no separate prefill graph).

## Contents (each `<arch>/` is a self-contained, flat artifacts root)
| file | role | size |
|------|------|------|
| `ternary-bonsai-1.7b-1024.json` | QHexRT manifest (llm family, `qwen3_generate` plan, YaRN) | — |
| `tbonsai17b_decn1k_w8.bin` | decode context binary (W8, GQA-native, maxctx 1024) | ~1.42 GB |
| `tbonsai17b_lmh_w8.bin` | tied lm-head context binary (W8) | ~0.31 GB |
| `tbonsai17b_embed_f16.bin` | host embedding table (fp16) | ~0.62 GB |
| `tokenizer.json` | tokenizer (Qwen2 BPE) | ~11 MB |

## Run (on-device, over adb) — pick the dir matching your chip
The QNN runtime libs (`libQnnHtp.so`, `libQnnSystem.so`, the HTP skel for your arch) come from the **QAIRT
SDK**, not this repo. Tool arg order is invariant: `<tool> <manifest> libQnnHtp.so libQnnSystem.so <root> …`.

```bash
ARCH=v81      # or v79 — MUST match your device's Hexagon arch (a bin won't load on the wrong arch)
hf download runanywhere/ternary_bonsai_1_7b_HNPU --include "$ARCH/*" --local-dir tbonsai
D=/data/local/tmp/tbonsai
adb push tbonsai/$ARCH/* $D/
# also stage into $D: qhx_generate + the QAIRT libs (libQnnHtp.so, libQnnSystem.so) + the HTP skel for $ARCH
adb shell "cd $D && export ADSP_LIBRARY_PATH='$D;/vendor/dsp/cdsp'; LD_LIBRARY_PATH=. \
  ./qhx_generate ternary-bonsai-1.7b-1024.json libQnnHtp.so libQnnSystem.so . 64 'The capital of France is'"
```

## Measured
- **v81 (SM8850, QAIRT 2.47.0.260601):** single-prompt **64/64 greedy-exact** vs HF fp16 (FP export gate 8/8);
  **~18.7 tok/s** decode (graph 45.8 + lm-head 7.4 ms/tok), prompt seed ~21.8 tok/s. W8 suite 4/6 strict,
  **6/6 coherent** (the 2 diffs are near-tie argmax-drift, not bugs).
- **v79 (SM8750):** compiled via the identical oracle-gated path (export gate 8/8); **on-device validation
  pending a v79 device** — expected to match v81 modulo W8 quant.

## Caveats
- **Arch-pinned:** `v79/` and `v81/` context binaries are NOT interchangeable — use the dir matching your
  device's Hexagon arch (`soc_model` 69 = v79, 87 = v81). A wrong-arch bin fails to *load* (it never runs).
- **W8 acceptance:** strict greedy_exact holds on the primary prompt but is best-effort across a suite (W8
  argmax-drift); ship an fp16 decode graph if you need strict whole-suite parity.
- Built with the **forge** pipeline (oracle-gated export → QAIRT W8 compile → on-device baseline gate).

---
_Precompiled for [QHexRT](https://github.com/RunanywhereAI/QHexRT). Base model:
[prism-ml/Ternary-Bonsai-1.7B-unpacked](https://huggingface.co/prism-ml/Ternary-Bonsai-1.7B-unpacked)._