# Reproducing these APEX quants

End-to-end recipe to rebuild both tiers from scratch. Everything is MIT-licensed
(see `NOTICE`).

## Pinned versions

- **llama.cpp** — commit `bbf4a8a` (build `b8833`) or any newer build that supports
  the `deepseek2` architecture, the `kimi-k2` pre-tokenizer, and
  `llama-quantize --tensor-type-file`. Binaries used: `llama-imatrix`,
  `llama-quantize`, `llama-perplexity`.
- **apex-quant** — commit `a445a12` (https://github.com/localai-org/apex-quant),
  for `scripts/generate_config.sh`. A copy of that script is bundled here as
  `generate_config.sh` for convenience.

## Inputs

- **Baseline (F16 GGUF):**
  [`gabriellarson/Moonlight-16B-A3B-Instruct-GGUF`](https://huggingface.co/gabriellarson/Moonlight-16B-A3B-Instruct-GGUF)
  → `Moonlight-16B-A3B-Instruct-F16.gguf`.
  (We quantize from a correctly-converted community F16 rather than converting the
  HF safetensors ourselves — Moonlight's Moonshot tiktoken tokenizer needs BPE
  merges that some llama.cpp converters drop. Verify any baseline with
  `tokenizer.ggml.merges` present and a coherent generation.)
- **Calibration data (for the imatrix):** Bartowski `calibration_datav3` —
  https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8

```bash
hf download gabriellarson/Moonlight-16B-A3B-Instruct-GGUF \
  Moonlight-16B-A3B-Instruct-F16.gguf --local-dir .
# save the gist as calibration_datav3.txt
```

## 1. Importance matrix (imatrix)

```bash
llama-imatrix -m Moonlight-16B-A3B-Instruct-F16.gguf \
  -f calibration_datav3.txt \
  -o Moonlight-16B-A3B-Instruct-F16.imatrix -ngl 999
```

## 2. Tensor-type configs

The final configs are included (`configs/moonlight_i-quality.txt`,
`configs/moonlight_handroll.txt`). To regenerate them:

```bash
# Base map: 27 layers, layer 0 is dense (first_k_dense_replace=1)
bash generate_config.sh --profile i-quality --layers 27 --dense-layers 1 \
  -o moonlight_i-quality.base.txt

# i-quality: patch in the real MLA attention tensors, drop non-existent ones
python patch_moonlight_config.py moonlight_i-quality.base.txt moonlight_i-quality.txt

# hand-roll: same, plus pin routed down projections to IQ4_NL (1408 dim isn't
# 256-divisible, so K-quants fall back to Q8_0; IQ4_NL is block-32)
python patch_moonlight_config.py moonlight_i-quality.base.txt moonlight_handroll.txt \
  --down-exps iq4_nl
```

## 3. Quantize (base type Q6_K for unlisted tensors: token_embd, output)

```bash
llama-quantize --tensor-type-file configs/moonlight_i-quality.txt \
  --imatrix Moonlight-16B-A3B-Instruct-F16.imatrix \
  Moonlight-16B-A3B-Instruct-F16.gguf \
  Moonlight-16B-A3B-Instruct-APEX-i-quality.gguf Q6_K

llama-quantize --tensor-type-file configs/moonlight_handroll.txt \
  --imatrix Moonlight-16B-A3B-Instruct-F16.imatrix \
  Moonlight-16B-A3B-Instruct-F16.gguf \
  Moonlight-16B-A3B-Instruct-APEX-handroll.gguf Q6_K
```

## 4. Evaluate (perplexity)

```bash
# NOTE: llama-perplexity prints the final estimate to STDERR — capture 2>&1.
for f in F16 APEX-i-quality APEX-handroll; do
  ppl=$(llama-perplexity -m Moonlight-16B-A3B-Instruct-$f.gguf \
        -f wiki.test.raw -ngl 999 --chunks 200 2>&1 \
        | grep -oP 'Final estimate: PPL = \K[0-9.]+')
  echo "$f  PPL=$ppl"
done
```

Expected (wikitext-2 test, 200×512 windows): F16 8.836 · i-quality 8.916 (+0.90%)
· hand-roll 8.959 (+1.38%).