piffie/Llama-3.2-3B-Instruct-Genie-Snapdragon-X2-Elite-v81

Llama 3.2 3B Instruct — Genie/QNN bundle for Snapdragon X2 Elite (Hexagon V81)

Built with Llama. This repo contains Meta's Llama-3.2-3B-Instruct compiled to Qualcomm Genie / QNN context binaries that run on the Hexagon NPU of the Snapdragon X2 Elite (soc_model 88, dsp_arch v81).

Why this exists

Qualcomm AI Hub and AnythingLLM currently ship LLM bundles compiled for the gen‑1 Snapdragon X Elite (Hexagon V73 / sc8380). Those do not run on the new X2 Elite (V81) — the serialized QNN context binaries are architecture‑specific (loading a V73 bundle on a V81 NPU hangs). At publish time there were no public V81 bundles, so this fills the gap for X2 owners.

Specs

Base model	meta-llama/Llama-3.2-3B-Instruct
Runtime	Qualcomm Genie (LLM)
Quantization	w4a16 (4‑bit weights, 16‑bit activations)
Target	Snapdragon X2 Elite, Hexagon V81 (soc_model 88)
Context length	4096
Compiled with (QAIRT)	2.45.0
Files	part{1,2,3}_of_3.bin (~2.6 GB) + genie_config.json, htp_backend_ext_config.json, tokenizer.json

How it was made

Compiled via qai-hub-models on Qualcomm AI Hub, targeting Snapdragon X2 Elite CRD, from the official base weights (no fine‑tuning — this is the stock model, just quantized + compiled for V81).

How to run (Python, on an X2 Elite)

from qai_appbuilder import GenieContext, QNNConfig
QNNConfig.Config("<path to qai_appbuilder/libs>", "Htp", 2, 0)
ctx = GenieContext("genie_config.json")   # run from the bundle directory
ctx.Query("<|begin_of_text|>...<|start_header_id|>assistant<|end_header_id|>\n\n",
          lambda tok, *_: print(tok.decode(errors="ignore"), end=""))

⚠️ Runtime requirements (important)

QNN/Genie is version‑coupled. To load these binaries you need a QAIRT runtime compatible with 2.45.x (e.g. a matching qai-appbuilder) and a current Hexagon NPU driver. A mismatch between the bundle's QAIRT version, the qai-appbuilder runtime, and the NPU driver can cause the load to hang rather than error. If it hangs, update the Hexagon NPU driver (Qualcomm Software Center) and align the runtime to QAIRT 2.45.x.

License

Governed by the Llama 3.2 Community License (see LICENSE/Meta's terms). Built with Llama; you must comply with Meta's Acceptable Use Policy.