File size: 3,373 Bytes
62ac664 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | # Reproducing these APEX quants
End-to-end recipe to rebuild both tiers from scratch. Everything is MIT-licensed
(see `NOTICE`).
## Pinned versions
- **llama.cpp** — commit `bbf4a8a` (build `b8833`) or any newer build that supports
the `deepseek2` architecture, the `kimi-k2` pre-tokenizer, and
`llama-quantize --tensor-type-file`. Binaries used: `llama-imatrix`,
`llama-quantize`, `llama-perplexity`.
- **apex-quant** — commit `a445a12` (https://github.com/localai-org/apex-quant),
for `scripts/generate_config.sh`. A copy of that script is bundled here as
`generate_config.sh` for convenience.
## Inputs
- **Baseline (F16 GGUF):**
[`gabriellarson/Moonlight-16B-A3B-Instruct-GGUF`](https://huggingface.co/gabriellarson/Moonlight-16B-A3B-Instruct-GGUF)
→ `Moonlight-16B-A3B-Instruct-F16.gguf`.
(We quantize from a correctly-converted community F16 rather than converting the
HF safetensors ourselves — Moonlight's Moonshot tiktoken tokenizer needs BPE
merges that some llama.cpp converters drop. Verify any baseline with
`tokenizer.ggml.merges` present and a coherent generation.)
- **Calibration data (for the imatrix):** Bartowski `calibration_datav3` —
https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8
```bash
hf download gabriellarson/Moonlight-16B-A3B-Instruct-GGUF \
Moonlight-16B-A3B-Instruct-F16.gguf --local-dir .
# save the gist as calibration_datav3.txt
```
## 1. Importance matrix (imatrix)
```bash
llama-imatrix -m Moonlight-16B-A3B-Instruct-F16.gguf \
-f calibration_datav3.txt \
-o Moonlight-16B-A3B-Instruct-F16.imatrix -ngl 999
```
## 2. Tensor-type configs
The final configs are included (`configs/moonlight_i-quality.txt`,
`configs/moonlight_handroll.txt`). To regenerate them:
```bash
# Base map: 27 layers, layer 0 is dense (first_k_dense_replace=1)
bash generate_config.sh --profile i-quality --layers 27 --dense-layers 1 \
-o moonlight_i-quality.base.txt
# i-quality: patch in the real MLA attention tensors, drop non-existent ones
python patch_moonlight_config.py moonlight_i-quality.base.txt moonlight_i-quality.txt
# hand-roll: same, plus pin routed down projections to IQ4_NL (1408 dim isn't
# 256-divisible, so K-quants fall back to Q8_0; IQ4_NL is block-32)
python patch_moonlight_config.py moonlight_i-quality.base.txt moonlight_handroll.txt \
--down-exps iq4_nl
```
## 3. Quantize (base type Q6_K for unlisted tensors: token_embd, output)
```bash
llama-quantize --tensor-type-file configs/moonlight_i-quality.txt \
--imatrix Moonlight-16B-A3B-Instruct-F16.imatrix \
Moonlight-16B-A3B-Instruct-F16.gguf \
Moonlight-16B-A3B-Instruct-APEX-i-quality.gguf Q6_K
llama-quantize --tensor-type-file configs/moonlight_handroll.txt \
--imatrix Moonlight-16B-A3B-Instruct-F16.imatrix \
Moonlight-16B-A3B-Instruct-F16.gguf \
Moonlight-16B-A3B-Instruct-APEX-handroll.gguf Q6_K
```
## 4. Evaluate (perplexity)
```bash
# NOTE: llama-perplexity prints the final estimate to STDERR — capture 2>&1.
for f in F16 APEX-i-quality APEX-handroll; do
ppl=$(llama-perplexity -m Moonlight-16B-A3B-Instruct-$f.gguf \
-f wiki.test.raw -ngl 999 --chunks 200 2>&1 \
| grep -oP 'Final estimate: PPL = \K[0-9.]+')
echo "$f PPL=$ppl"
done
```
Expected (wikitext-2 test, 200×512 windows): F16 8.836 · i-quality 8.916 (+0.90%)
· hand-roll 8.959 (+1.38%).
|