v75 (Galaxy S24 / SM8650, soc_model 57): Llama-3.1-Nemotron-Nano-8B 8-shard W8 bundle — export 8/8 (runtime needs >8GB RAM)
#1
by flying101 - opened
Adds the Hexagon v75 flat bundle under v75/ (Snapdragon 8 Gen 3 class).
- Files: nemotron-nano-8b.json, nemotronnano8b_decode_p0..p7_w8.bin (8-shard W8 decode,
0.9G each), nemotronnano8b_lmhead_w8.bin, nemotronnano8b_embed_f16.bin, tokenizer.json (8.1G total) - Precision: 8-shard W8 decode + W8 lm-head + fp16 embed (large-LLM decode-split path, host-op llama_generate_sharded)
- Compile: QAIRT 2.47.0, dsp_arch v75, soc_model 57; export gate decode 8/8 match HF greedy
- Device note: all 8 decode shards load on v75, but the ~8G resident W8 weights exceed the 8GB Galaxy S24 RAM at runtime (adb drop). Runs on higher-RAM v75 devices (v81 measured ~6GB RSS). Published for arch parity with v79/v81.
flying101 changed pull request status to open
sanmonga22 changed pull request status to merged