sanmonga22 commited on
Commit
f850664
·
verified ·
1 Parent(s): b9d2499

Sanitize public model card metadata

Browse files
Files changed (1) hide show
  1. README.md +11 -53
README.md CHANGED
@@ -1,60 +1,18 @@
1
  ---
2
- license: llama3.2
3
  tags:
4
- - qhexrt
5
- - llama
6
- - hexagon
7
- - npu
8
- - qnn
9
- base_model: meta-llama/Llama-3.2-1B
10
- pipeline_tag: text-generation
11
- language:
12
- - en
13
  ---
14
 
15
- # Llama-3.2-1B — QHexRT NPU bundle (Hexagon v79)
16
 
17
- Precompiled **Llama-3.2-1B** for the **QHexRT** runtime on Qualcomm Hexagon **v79** (Snapdragon 8 Elite /
18
- SM8750, e.g. Galaxy S25). Runs **entirely on the NPU**: the full 16-layer transformer **and** the lm-head
19
- execute on HTP/HMX via the `llama_generate` host-op (batched prefill + resident-KV decode); the host does
20
- only embed lookup, llama3-scaled RoPE, the cache mask, and sampling. Device-validated, coherent output.
21
 
22
- **Beats Genie.** On the same device, QHexRT decode is **~16 tok/s vs Genie's 14.69** (fp16), and prefill
23
- **1983 tok/s vs 1538** — because QHexRT runs a prepared QNN context binary with an **in-graph NPU lm-head**
24
- (the host lm-head was the old bottleneck) and **W8A16** weights (half fp16's memory bandwidth).
25
 
26
- ## Contents (`v79/`)
27
- | file | what | size |
28
- |------|------|------|
29
- | `llama-3.2-1b.json` | QHexRT manifest (llm family, `llama_generate` plan) | — |
30
- | `llama_full_wqo_o3.bin` | **W8A16 batched prefill** graph (AR=128) | 982 MB |
31
- | `llama_dec_wqo_o3.bin` | **W8A16 decode** graph (MAXCTX=512, GQA 32q/8kv, head_dim 64) | 981 MB |
32
- | `llama_lmhead.bin` | **NPU lm-head** graph — `hidden[1,2048] → logits[1,128256]`, **fp16** on HMX | 526 MB |
33
- | `llama_embed_f16.bin` | token embedding table `[128256,2048]` f16 (tied; host lookup) | 525 MB |
34
- | `tokenizer.json` | Llama-3 tokenizer (vocab 128256) | — |
35
-
36
- ≈ **3 GB** on disk; ~2.5 GB peak device RSS.
37
-
38
- ## Run (QHexRT CLI)
39
- ```bash
40
- hf download runanywhere/llama3_2_1b_HNPU --local-dir llama3_2_1b_HNPU
41
- adb push llama3_2_1b_HNPU/v79 /data/local/tmp/wq/llama
42
- adb shell "cd /data/local/tmp/wq && export ADSP_LIBRARY_PATH='/data/local/tmp/wq/dsp;/data/local/tmp/wq;/vendor/dsp/cdsp'; \
43
- LD_LIBRARY_PATH=. ./qhx_generate llama/llama-3.2-1b.json libQnnHtp.so libQnnSystem.so llama 64 'The capital of France is'"
44
- # -> "Paris. The Eiffel Tower is located in Paris. ..." (decode ~16 tok/s, prefill ~1983 tok/s)
45
- ```
46
-
47
- ## Performance (measured, v79 / S25)
48
- | metric | QHexRT | Genie (fp16) |
49
- |--------|--------|--------------|
50
- | decode | **~16 tok/s** | 14.69 |
51
- | prefill | **1983 tok/s** | 1538 |
52
-
53
- ## Notes
54
- - Arch: **v79 only** — context binaries are dsp-arch-pinned.
55
- - No custom op-package — pure-native HTP graphs (W8A16 matmuls on HMX).
56
- - **Prompt ≤128 tokens** (the batched-prefill graph is AR=128); generation extends to MAXCTX=512.
57
- - Precision: **W8A16** decode/prefill + **fp16** lm-head. W4A16 was tried but collapses a 1B model's
58
- coherence (uniform 4-bit is too coarse here); W8A16 is the coherent sweet spot and still beats Genie.
59
- - Source: [`meta-llama/Llama-3.2-1B`](https://huggingface.co/meta-llama/Llama-3.2-1B), compiled with
60
- QAIRT 2.45 for `qualcomm-snapdragon-8-elite-for-galaxy`.
 
1
  ---
2
+ license: "llama3.2"
3
  tags:
4
+ - "hnpu"
5
+ - "hexagon"
6
+ - "npu"
7
+ - "llm"
8
+ base_model: "meta-llama/Llama-3.2-1B"
9
+ pipeline_tag: "text-generation"
 
 
 
10
  ---
11
 
12
+ # llama3 2 1b HNPU
13
 
14
+ Prebuilt HNPU artifacts for [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B), a public text-generation model.
 
 
 
15
 
16
+ For model behavior, license, intended use, and limitations, see the [upstream model card](https://huggingface.co/meta-llama/Llama-3.2-1B).
 
 
17
 
18
+ Artifacts are architecture-pinned. Available artifact directories: `v79/`, `v81/`.