Duplicate from canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP

Browse files

Co-authored-by: Paul Cozzolino <pastapaul@users.noreply.huggingface.co>

Files changed (12) hide show

.gitattributes +35 -0
README.md +428 -0
config.json +1163 -0
generation_config.json +9 -0
model-00001-of-00004.safetensors +3 -0
model-00002-of-00004.safetensors +3 -0
model-00003-of-00004.safetensors +3 -0
model-00004-of-00004.safetensors +3 -0
model.safetensors.index.json +0 -0
recipe.yaml +61 -0
tokenizer.json +0 -0
tokenizer_config.json +14 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,35 @@

+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,428 @@

+---
+license: mit
+base_model: deepseek-ai/DeepSeek-V4-Flash
+base_model_relation: quantized
+language:
+- en
+- zh
+library_name: vllm
+pipeline_tag: text-generation
+tags:
+- deepseek
+- deepseek_v4
+- compressed-tensors
+- w4a16
+- gptq
+- fp8
+- mtp
+- speculative-decoding
+- mixture-of-experts
+- moe
+- vllm
+---
+# canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP
+W4A16 INT4 routed experts + FP8 block 128×128 attention + **BF16 Multi-Token Prediction (MTP) draft head retained** — the first DeepSeek-V4-Flash quantization that ships a working MTP block, giving ~1.5× speculative decoding (spec-decode) speedup at bs=1 with no quality cost. Extends the [W4A16-FP8 predecessor](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8) by patching the `transformers` calibration path so the MTP block survives the load.
+## TL;DR
+| | |
+|---|---|
+| **Recommended hardware** | RTX PRO 6000 Blackwell at **TP=2 (2 GPUs/replica)** or **TP=4 (4 GPUs/replica)** — both validated · or 8× H200 TP=2 |
+| **Quality** | GSM8K 93.71% (8-shot strict); HumanEval 84.76% pass@1; MMLU 86.88% |
+| **Throughput** | RTX PRO 6000 **98.83 @ TP=2** / **107.32 @ TP=4** at bs=1; 88.35 on H200 TP=2 |
+| **MTP acceptance** | 89% calibrated workload / 70% on random prompts at bs=1 k=1 |
+| **Spec-decode speedup** | **1.49× at bs=1, k=1** (TPOT 6.02 ms vs 8.93 ms, same artifact) |
+| **Differentiator** | First V4-Flash W4A16 quant where MTP survives the calibration load; `transformers` 5.8.1 silently strips MTP keys by default |
+## Family / related artifacts
+| Repo | Role | Relation to this artifact |
+|---|---|---|
+| [`canada-quant/DeepSeek-V4-Flash-W4A16-FP8`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8) | predecessor | Same W4A16 + FP8 recipe; MTP dropped at load (the bug this artifact fixes) |
+| [`canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP) | sibling | Same MTP-retention pattern; NVFP4 routed experts instead of W4A16 (Blackwell-native) |
+| [`canada-quant/DeepSeek-V4-Pro-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Pro-NVFP4-FP8-MTP) | larger sibling | V4-Pro at NVFP4 + MTP, B300-only deployment |
+| [`RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8`](https://huggingface.co/RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8) | upstream reference | Original NVFP4 recipe (no MTP — same silent-drop bug) |
+## Why this exists
+The [W4A16-FP8 predecessor](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8) and [`RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8`](https://huggingface.co/RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8) both **drop the MTP block** because `transformers` 5.8.1's `DeepseekV4PreTrainedModel` declares:
+```python
+_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
+```
+which silently filters every `mtp.*` tensor at `from_pretrained` time — without warning, without error. Calibration pipelines that go through `from_pretrained` produce quantized main weights paired with an absent MTP block; serving falls back to plain decode, losing the ~1.5–2× spec-decode speedup that V4-Flash's architecture provides.
+This artifact bypasses the silent drop, runs the full 8-rank GPTQ calibration on a 768-sample corpus against the main routed experts, preserves the MTP block unquantized in BF16, and produces a serving artifact where speculative decoding actually fires.
+## Architecture & precision
+### Base model
+| Property | Value |
+|---|---|
+| Total parameters | ~284 B (~13 B active per token) |
+| Decoder layers | 43 |
+| Routed experts / layer | 256 (top-K = 6) |
+| Hidden size | 4096 |
+| Base BF16 size | ~543 GB |
+| Quantized size | **159 GB** (+3 GB vs predecessor for the BF16 MTP block) |
+### Component precisions
+| Component | Format | Method |
+|---|---|---|
+| Routed experts (256 × 43 layers × 3 projections) | W4A16 INT4, group_size=128, symmetric | GPTQ via llm-compressor, 768 calibration samples |
+| Attention path (`wq_a`, `wq_b`, `wkv`, `wo_a`, `wo_b`, indexer, compressor) | FP8_BLOCK 128×128 | Dynamic scales, `scale_fmt=ue8m0` |
+| **MTP block (`mtp.0.*`)** | **BF16** | **Excluded from quantization, preserved verbatim** |
+| HC plumbing (`hc_attn_*`, `hc_ffn_*`, `hc_head_*`), `attn_sink`, `ffn.gate.bias`, indexer/compressor `ape` | FP32 | Restored post-save from BF16 source (see [Upstream contributions](#upstream-contributions-filed-during-this-work)) |
+| `head.weight` (LM head) | FP32 | Upcast from BF16 to match sibling artifact's MTP loader path |
+| Embeddings (`embed.weight`, `mtp.0.emb.tok_emb.weight`) | BF16 | Source dtype preserved |
+## Hardware validated
+| Platform | SM | HBM/GPU | Interconnect | TP | Role |
+|---|---|---|---|---|---|
+| 8× NVIDIA H200 SXM5 | 9.0a | 141 GB HBM3e | NVLink | 2 (4× replicas) | Calibration + initial benchmarks (`p5en.48xlarge`) |
+| 4× NVIDIA RTX PRO 6000 Blackwell Server Edition | 12.0, sm_120 | 96 GB HBM | PCIe | **TP=2 (2 GPUs, 2 replicas on a 4-GPU box) or TP=4 (4 GPUs, 1 replica)** | Workstation Blackwell deployment + $/token sweet spot |
+Same artifact, no weight changes between SKUs. Both validated cuda graphs ON.
+## Benchmarks
+All numbers from the same artifact, vLLM HEAD `50d9dd902` + 4 patches cherry-picked (PRs #43248 / #43288 / #43290 / #43319).
+### Quality
+Sampling: greedy, temperature 0. Methodology disclosed per row.
+| Benchmark | Setting | This artifact | Predecessor (W4A16-FP8, no MTP) | RedHat (NVFP4-FP8, no MTP) | Delta |
+|---|---|---|---|---|---|
+| GSM8K | 8-shot, strict-match | **93.71% ± 0.67** | 95.07% (RTX PRO 6000) / 95.45% (Spark) | 91.0% (self-reported) | -1.28 pts vs predecessor (within 1 SE) |
+| GSM8K | 8-shot, flexible-extract | 93.63% ± 0.67 | 95.37% (Spark) | — | within SE |
+| MMLU | 5-shot | **86.88% ± 0.27** | 87.27% (H200) | — | -0.39 pts (within SE) |
+| MMLU-Pro | 5-shot, 12k prompts, custom-extract | **71.28% ± 0.40** | — | — | sibling NVFP4-FP8-MTP scored 81.13% on B300 — expected gap given W4A16 has more quant noise than NVFP4 on knowledge-heavy harder benchmarks |
+| HumanEval | 0-shot pass@1, `--confirm_run_unsafe_code` | **84.76% ± 2.82** | 80.49% (corrected, see predecessor card "Changes") | — | +4.27 pts vs corrected predecessor number |
+| AIME 2024 | 30 problems, thinking=high, c=4, max_tokens=64K | **29/30 (96.7%)** ✓ verified 2026-05-29 in fresh Docker (TP=4 RTX PRO 6000) | — | — | the prior 30.0% number was a scoring artifact (see footnote); proper chat-template thinking=high + max_tokens at the model-len cap (so reasoning isn't truncated) returns the right answer. TP=2 same config: 27/30 (90.0%). |
+| GSM8K-50 chat-mode cross-check (RTX PRO 6000 TP=4, 2026-05-24 post-shipping-fix) | greedy, no thinking, concurrency=1 | **44/50 = 88.0%** | — | — | matches Card B sibling's 88% strict TP=2 / 90% strict TP=4 on the same hardware — confirms dequant'd artifact preserves quality |
+| IFEval prompt-strict | chat-template, no thinking | **TBD**² | — | — | not yet measured cleanly on this build |
+| chat-smoke (quick / quality / coding) | harness | 4/4 · 4/4 · 2/2 | 4/4 · 4/4 · 2/2 | — | match |
+| toolcall15 | 1 round, 30 points | 24/30 (80%) | 26/30 (87%) | — | -2 pts — see [Honest limitations](#honest-limitations) |
+¹ The prior 30.0% AIME number was an lm-eval-harness `aime24` task artifact — completions-mode prompt (no chat template), `exact_match` scorer on a thinking-mode model whose answers are wrapped in `<think>…</think>` + `\boxed{N}`. The scorer matched the literal answer string and missed virtually every correct response. A 1-shot smoke under proper chat-templated thinking=high methodology returned **2024-II-4: pred=33, exp=33, correct** in 2072 completion tokens — model behavior is correct. Full 30-problem re-bench attempted 2026-05-24 on this RTX PRO 6000 box hit a reproducible CUDA illegal memory access (`Worker_TP2: torch.AcceleratorError`) under any concurrent thinking-mode load (cuda graphs and `--enforce-eager` both crash; concurrency=4 dies at ~11 min, concurrency=8 dies at ~90 s). Single-shot inference works. Re-bench deferred to H200 with `jasl/vllm@ds4-sm120-experimental@abad5dc71` (the build the original Card D H200 numbers used).
+² IFEval re-bench attempted 2026-05-24 hit the same RTX PRO 6000 stability issue. Deferred alongside AIME.
+### Throughput
+`vllm bench serve` random 256-in / 256-out, MTP-spec `num_speculative_tokens=1` (k=1 cap on this build — see [Honest limitations](#honest-limitations)), cuda graphs ON.
+| Hardware | TP | bs=1 output tok/s | bs=1 TPOT median | bs=4 output tok/s | bs=16 output tok/s | MTP acceptance @ bs=1 |
+|---|---|---|---|---|---|---|
+| 8× H200 | 2 (per replica) | 88.35 | **6.02 ms** | 138.80 | 367.13 | 89% calibrated / 70% random |
+| 4× RTX PRO 6000 box | TP=2 (per replica, 2 replicas fit) | **98.83** | 8.55 ms | 219.53 | 482.61 | 71% |
+| 4× RTX PRO 6000 box | TP=4 (single replica) | **107.32** | 7.77 ms | 221.52 | **584.04** | 68% |
+Per-replica, RTX PRO 6000 wins output throughput at every batch size; H200 still wins per-token TPOT median.
+### MTP draft-token acceptance per workload
+Same artifact, bs=1, k=1.
+| Workload | Prompts | Accepted / emitted | Acceptance |
+|---|---|---|---|
+| Random 256-token prompts (200 samples) | random | 21024 / 30058 | **69.94%** |
+| Code, raw completion (15 short signature+docstring prompts) | code-raw | 1847 / 1988 | **92.91%** |
+| Chat-templated prose (15 prompts) | chat-prose | 1946 / 2376 | **81.90%** |
+| Raw natural language (15 continuation prompts) | nl-raw | 1745 / 2086 | **83.65%** |
+Spec-decode wins at low concurrency (single-user interactive). At bs≥4 the verifier is already filling its batch lane, so extra verifier passes add overhead without saving wall-clock — matches the sibling artifact's framing of bs=1 as the headline operating point.
+### Cost per output token (node-level)
+Boxes priced for cloud-rented hardware. Single-replica numbers measured; multi-replica totals are linear extrapolation.
+| Box | Replicas | bs=1 total tok/s | bs=16 total tok/s | $/h | $/(1000 tok/h) at bs=1 |
+|---|---|---|---|---|---|
+| `p5en.48xlarge` (8× H200) | 4× TP=2 | ~353 | ~1468 | $98 | **$278** |
+| `g7e.24xlarge` (4× RTX PRO 6000) | 2× TP=2 | ~198 | ~965 | $19.92 | **$101** |
+| `g7e.24xlarge` (4× RTX PRO 6000) | 1× TP=4 | 107.32 | 584.04 | $19.92 | $186 |
+**At bs=1 (interactive), RTX PRO 6000 2×TP=2 is ~2.7× cheaper than H200 4×TP=2.** At bs=16 the gap narrows because H200's per-replica throughput scales better with batch — H200 wins absolute throughput when you can fill it; RTX wins on $/token unless you genuinely need >1500 tok/s aggregate output.
+### Cross-validation: 2026-05-29 fresh Docker on RTX PRO 6000 ✓
+> **Hardware coverage:** all numbers below are from **RTX PRO 6000 Blackwell Server Edition (SM 12.0a)** on a Brev `g7e.24xlarge`. The image is **expected to work on Workstation Edition (same SM 12.0a, same Marlin native cubins, same model + serve path)** but we have not directly verified it ourselves. Reference TP=2 Workstation numbers from jasl's bench harness ([`baselines/20260512_sm120_deployment_1c20f1a6d`](https://github.com/jasl/vllm-ds4-sm120-harness/tree/main/baselines/20260512_sm120_deployment_1c20f1a6d)) confirm the underlying stack runs on Workstation. Expect a 5-15% throughput delta from clock/memory-bandwidth differences. If anything misbehaves on Workstation Edition, open an issue at [the repo](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp).
+Full bench matrix on `canada-quant/dsv4-w4a16-rtxpro6000:v1` (the [HF-published Docker image](https://huggingface.co/datasets/canada-quant/dsv4-flash-w4a16-rtxpro6000-image), built from `jasl/vllm@27fd665b` + canada-quant BF16-MTP cherry-pick + Marlin MoE `c_tmp`/workspace patches). All AIME runs at `max_tokens = max_model_len - 500 = 65036` so reasoning runs to natural stop:
+| AIME-2024 thinking-mode sweep (c=4, n=30) | TP=2 (max_num_seqs=4) | TP=4 (max_num_seqs=16) |
+|---|---|---|
+| chat (no think) | 18/30 · MTP 95.78% · 53m | 19/30 · MTP 93.06% · 5m |
+| **thinking-high** | **27/30 · MTP 91.97% · 152m** | **29/30 · MTP 91.01% · 13m** |
+| thinking-max | 24/30 · MTP 92.52% · 177m | 27/30 · MTP 91.68% · 26m |
+| AIME-2024 single-shot reference (c=1, thinking-high, n=30) | TP=2 | TP=4 |
+|---|---|---|
+| c=1 high | 27/30 · MTP 91.68% · 48m | **28/30 · MTP 90.76% · 41m** |
+| GSM8K (n=50, 8-shot) | TP=2 | TP=4 |
+|---|---|---|
+| flexible-extract | 45/50 (90.0%) | 43/50 (86.0%) |
+| strict-match | 42/50 (84.0%) | 40/50 (80.0%) |
+| Throughput random 256/256 (single replica, MTP on) | TP=2 tok/s @ TPOT p50 | TP=4 tok/s @ TPOT p50 |
+|---|---|---|
+| bs=1 | 95.2 @ 8.05 ms | **108.1 @ 7.32 ms** |
+| bs=4 | 40.6 @ 83.18 ms | 104.3 @ 11.31 ms |
+| bs=8 | 45.7 @ 79.02 ms | **433.2 @ 16.44 ms** (sweet spot) |
+| bs=16 | 34.9 (capped by max_num_seqs=4) | 164.3 (scheduler thrash) |
+| Throughput random 1024/1024 | TP=2 | TP=4 |
+|---|---|---|
+| bs=1 | 30.7 tok/s | 138.1 tok/s |
+| bs=4 | 45.1 tok/s | 363.7 tok/s |
+**Headlines from this run:**
+- **Zero CUDA illegal-memory-access** in 240 AIME thinking-mode problems across c=4 chat/high/max + c=1 high on both TP=2 and TP=4 = the Marlin MoE concurrent-decode race is fixed by the `c_tmp` clamp removal in PR [vllm#43730](https://github.com/vllm-project/vllm/pull/43730) (which is baked into the v3 image via `jasl/vllm@27fd665b`).
+- **TP=4 is 7-12× faster** than TP=2 at AIME (chat 53m→5m, high 152m→13m, max 177m→26m). MoE expert sharding across 4 GPUs decisively wins.
+- **Thinking-max regresses correctness AND triples wall** (TP=4: high 29/30 in 13m vs max 27/30 in 26m). The artifact's sweet spot is `reasoning_effort=high`.
+- **MTP holds 91-93%** across all thinking modes and TP configs — the BF16-retained draft head is doing its job everywhere.
+Raw JSON + per-bench logs in the [reproduction repo](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/tree/main/benchmarks/rtxpro6000_docker_v3).
+#### Tuning attempts that DID NOT win on TP=4 Server (documenting so you don't repeat them)
+We A/B-tested adopting jasl's TP=2 Workstation env tunings at TP=4 Server — none of them transferred. Stick with the v3 image defaults:
+| Change from defaults | Result | Why |
+|---|---|---|
+| `num_speculative_tokens=2` (jasl's `deepseek_mtp` k=2 default) | **−86% bs=8** (433 → 60 tok/s) | k=2 doubles main-model forward cost; at TP=4 the all-reduce overhead exceeds the ~1.5 tokens-per-draft acceptance gain that's net-positive at TP=2 |
+| `--enable-expert-parallel` (jasl recommends) | similarly bad combined with k=2 | TP=4 all-to-all expert-gather is expensive |
+| `VLLM_TRITON_MLA_SPARSE_QUERY_CHUNK_SIZE=512` + `..._TOPK_CHUNK_SIZE=512` (jasl's chunk-size tunings) | **CUDA illegal memory access at cudagraph capture** | Tuned for SM 12.0a c128a Workstation single-request prefill; exceed safe limits at TP=4 Server |
+| `--no-enable-flashinfer-autotune` (jasl recommends) | **−74% bs=8** (433 → 111 tok/s) | Triton block-FP8 autotune is load-bearing at TP=4 — disabling locks in default tile sizes that don't match the 4-GPU shape |
+| `--gpu-memory-utilization 0.985` (jasl recommends) | crash potential combined with sparse-MLA env | 0.95 is the safe value the v3 image ships with at TP=4 |
+The image's defaults are the optimal config for TP=4 RTX PRO 6000 Server Edition as of 2026-05-29. If you're deploying on **TP=2 Workstation Edition**, jasl's reference config ([`sm120_tp2_serve.env.example`](https://github.com/jasl/vllm-ds4-sm120-harness/blob/main/configs/sm120_tp2_serve.env.example)) is the right starting point — it was tuned on that exact hardware.
+## Quick start
+### RTX PRO 6000 Blackwell — Docker (recommended)
+The pre-built [`canada-quant/dsv4-w4a16-rtxpro6000:v1`](https://huggingface.co/datasets/canada-quant/dsv4-flash-w4a16-rtxpro6000-image)
+image bakes the full 13-layer recipe (`jasl/vllm@27fd665b` + canada-quant BF16
+MTP cherry-pick + Marlin MoE c_tmp/workspace patches + `cute.arch.fmin` shim).
+~3-5 min from `docker load` to a working endpoint on a g7e.24xlarge.
+```bash
+# 1. Pull the image tarball (~14 GB compressed)
+hf download canada-quant/dsv4-flash-w4a16-rtxpro6000-image \
+    --include "*.tar.gz" --local-dir .
+docker load < dsv4-w4a16-rtxpro6000-v1.tar.gz
+# 2. Cache the W4A16 model onto NVMe (~159 GB, ~1-2 min via xet on Brev)
+HF_HOME=/opt/dlami/nvme/hf-cache hf download \
+    canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP
+# 3. Pull the serve helper
+git clone https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp.git
+cd dsv4-flash-w4a16-fp8-mtp
+# 4. Serve TP=2 (or TP=4 with --gpus all -e TP=4 -e MAX_NUM_SEQS=16)
+docker run -d --gpus '"device=0,1"' --name dsv4-w4a16-serve \
+    --shm-size=16g --ipc=host -p 8000:8000 \
+    -v /opt/dlami/nvme/hf-cache:/root/.cache/huggingface \
+    -v $(pwd)/scripts:/workspace/scripts:ro \
+    -e TP=2 -e MAX_NUM_SEQS=4 -e MAX_MODEL_LEN=65536 -e GPU_MEM_UTIL=0.95 \
+    canada-quant/dsv4-w4a16-rtxpro6000:v1 \
+    bash /workspace/scripts/serve_rtx6000pro_w4a16.sh
+# 5. Wait for /v1/models (~3-5 min model load + cudagraph capture)
+until curl -sf http://127.0.0.1:8000/v1/models >/dev/null; do sleep 5; done
+# 6. Run the full bench matrix (AIME chat/high/max + GSM8K + throughput)
+docker exec dsv4-w4a16-serve bash -c \
+    "TAG=tp2_64k MAX_MODEL_LEN=65536 bash /workspace/scripts/bench_matrix.sh"
+```
+### RTX PRO 6000 Blackwell — from-source install (advanced)
+```bash
+# 1. Bootstrap vLLM (~25 min for source build)
+git clone https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp.git
+cd dsv4-flash-w4a16-fp8-mtp
+bash scripts/bootstrap_rtx6000pro.sh
+# 2. Extra pins
+source ~/venv-serve/bin/activate
+pip install --quiet "flashinfer-python==0.6.8.post1" "flashinfer-cubin==0.6.8.post1" \
+    "numba==0.65.0" "tilelang==0.1.9" "apache-tvm-ffi==0.1.9" "fastsafetensors>=0.2.2"
+# 3. Apply patches
+python scripts/patch_v4_forcausal_packed_mapping.py "$(python -c 'import vllm; print(vllm.__path__[0])')"
+python scripts/patch_mtp_packed_mapping.py        "$(python -c 'import vllm; print(vllm.__path__[0])')"
+python scripts/patch_nvidia_attn_scale.py         "$(python -c 'import vllm; print(vllm.__path__[0])')"
+bash   scripts/patch_wo_a_bf16_path.sh             "$(python -c 'import vllm; print(vllm.__path__[0])')"
+# 4. Download artifact (159 GiB) — already dequant'd in-artifact as of 2026-05-24,
+#    no local preprocessing step required.
+hf download canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP \
+    --local-dir /scratch/weights/w4a16-fp8-mtp-gptq
+# 5. Serve TP=2 (or TP=4 with 0,1,2,3)
+CUDA_VISIBLE_DEVICES=0,1 bash scripts/serve_rtx6000pro.sh \
+    /scratch/weights/w4a16-fp8-mtp-gptq 8000 2
+```
+**Required runtime env vars on SM 12.x (already set inside `serve_rtx6000pro.sh` but worth knowing):**
+```bash
+export VLLM_TRITON_MLA_SPARSE=1
+export VLLM_TRITON_MLA_SPARSE_HEAD_BLOCK_SIZE=4
+export VLLM_USE_FLASHINFER_SAMPLER=0
+```
+Without `VLLM_TRITON_MLA_SPARSE_HEAD_BLOCK_SIZE=4` the sparse-MLA Triton kernel can crash during warmup with `RuntimeError: Triton Error [CUDA]: an illegal memory access` in `_dequantize_and_gather_k_kernel`. The FlashInfer sampler is also broken on `TORCH_CUDA_ARCH_LIST=12.0a` — fall back to PyTorch-native via `VLLM_USE_FLASHINFER_SAMPLER=0`.
+### H200
+```bash
+vllm serve canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP \
+    --tensor-parallel-size 2 \
+    --kv-cache-dtype fp8 --block-size 256 \
+    --max-model-len 4096 \
+    --gpu-memory-utilization 0.80 \
+    --no-enable-prefix-caching \
+    --tokenizer-mode deepseek_v4 \
+    --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
+    --reasoning-parser deepseek_v4 \
+    --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
+    --trust-remote-code
+```
+## Quantization recipe
+| Property | Value |
+|---|---|
+| Dataset | `HuggingFaceH4/ultrachat_200k` (V4 chat template) |
+| Samples | 768 |
+| Max sequence length | 512 |
+| Per-rank batch size | 4 |
+| Calibration hardware | 8× NVIDIA H200 (`p5en.48xlarge`) |
+| Walltime | ~15.4h (15.09h oneshot + ~16 min save) |
+| Per-subgraph cadence | ~20 min/subgraph × 44 subgraphs (43 MoE + 1 MTP no-op) |
+Calibration recipe identical to the [W4A16-FP8 predecessor](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8) with one change: the modeling class is patched to remove `mtp.*` from `_keys_to_ignore_on_load_unexpected` before `from_pretrained`, so the MTP block survives the load and is written back to the artifact at BF16.
+## vLLM build
+### Common patches (all platforms)
+| PR | Purpose | Status |
+|---|---|---|
+| [`vllm-project/vllm#43248`](https://github.com/vllm-project/vllm/pull/43248) | `bool()` wrap on `is_static_input_scheme` | open |
+| [`vllm-project/vllm#43288`](https://github.com/vllm-project/vllm/pull/43288) | `.get("scale_fmt", "ue8m0")` on missing key + BF16 `getattr` follow-up | open |
+| [`vllm-project/vllm#43290`](https://github.com/vllm-project/vllm/pull/43290) | `weight_scale_inv`-or-`weight_scale` fallback (attention) | open |
+| [`vllm-project/vllm#43319`](https://github.com/vllm-project/vllm/pull/43319) | MTP-quant-detect from safetensors header + BF16 `wo_a` fallback path | open |
+### RTX PRO 6000 Blackwell (SM 12.0) only
+| Patch | Purpose |
+|---|---|
+| `packed_modules_mapping` on `DeepseekV4ForCausalLM` + `DeepSeekV4MTP` | Required as of `ds4-sm120-experimental@abad5dc71` |
+| BF16 `wo_a` path for MTP block | Static `weight.dtype == bfloat16` check (dynamo-safe) |
+| `--disable-custom-all-reduce` | No NVLink between RTX PRO 6000 boards |
+| CMakeLists `USE_SABI 3.11` removal | For Python 3.10 |
+(Previously this list also required a `compressor/indexer FP8 → BF16 dequant preprocess` step run against the local artifact. As of 2026-05-24 the dequant is baked into the published artifact — see [Changes](#changes).)
+H200 deployments need only the four common patches.
+## Honest limitations
+1. **k=1 cap on spec-decode** — current vLLM build limits `num_speculative_tokens` to 1 due to DeepGemm kernel assertion `next_n == 1 or next_n == 2` in `smxx_fp8_fp4_paged_mqa_logits.hpp:233`. vLLM passes `next_n = num_speculative_tokens + 1`, so practical k is 1. The `FLASHINFER_MLA_SPARSE` attention backend hits the same kernel-side assertion. With the assertion relaxed, expect bs=1 speedup to rise from 1.49× to ~1.85× (matching sibling NVFP4 artifact's k=2 published number).
+2. **Concurrent thinking-mode workloads on RTX PRO 6000 produce token-corrupted output** — under concurrency ≥ 2 with thinking=high (long-decode workloads like AIME), the Marlin W4A16 MoE decode kernel on SM 12.0 produces token-stream corruption (CJK / Cyrillic / garbled ASCII spliced into the model's reasoning trace). The same hardware + same vLLM build serving the **NVFP4 sibling** ([`canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP) via `flashinfer_trtllm` MoE) is essentially clean on the same workload (1/30 vs 14/30 corrupted at c=4 thinking). The bug is **specific to the W4A16 + Marlin MoE decode path on SM 12.0**. Investigation isolated through 7 controlled tests (sparse-MLA topk-chunk size, MTP-off, matmul_decode-off, eager-mode, concurrency sweep, NVFP4 vs W4A16 path comparison). **Workaround on RTX PRO 6000:** for batched thinking-mode workloads, serve the NVFP4 sibling artifact instead. For sequential (c=1) thinking-mode or any batched chat-mode (no thinking), this W4A16-MTP artifact works cleanly (GSM8K-20 chat-mode sequential = 20/20 = 100%, MTP draft acceptance 92.46%). Full debug log + reproducible benches: [`docs/findings/sm12x_token_corruption_2026_05_24.md`](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/blob/main/docs/findings/sm12x_token_corruption_2026_05_24.md). Filed upstream as [`jasl/vllm#12`](https://github.com/jasl/vllm/issues/12).
+3. **toolcall15 -2 pts vs predecessor** — model-routing regressions on chain-completion (TC-07 stopped mid-chain to ask a clarifying question) and multi-tool extraction (TC-06 returned both translations as content text instead of routing two `translate` calls). Quality-wise the model completes the underlying intent; the harness scores tool-call-protocol fidelity, not task completion. Not a parser issue (confirmed by replay through `--tool-call-parser deepseek_v4`).
+4. **GSM8K -1.3 pts** vs predecessor's 8-shot strict-match — within one SE, but technically below. Likely calibration-set sensitivity rather than recipe drift (recipe is identical, hardware differs).
+5. **NVFP4 native kernels on RTX PRO 6000 not auto-selected** — even though `csrc/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu` exists in upstream vLLM, the backend selector doesn't pick it ([`vllm-project/vllm#31085`](https://github.com/vllm-project/vllm/issues/31085)). Until that lands, the sibling NVFP4 artifact on this hardware would route through Marlin too. This artifact's W4A16 path is the tested choice for RTX PRO 6000.
+## Reproduction
+Full pipeline at [`canada-quant/dsv4-flash-w4a16-fp8-mtp`](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp). From a fresh 8× H200 box:
+```bash
+# Phase 0 — bootstrap (venv-calib + venv-serve + vendor + apply patches)
+bash scripts/bootstrap_p5en_h200.sh
+# Phase 1 — download upstream + dequant to BF16-MTP source (~30 min, ~660 GB)
+bash scripts/phase1_dequant.sh
+# Phase 2 — GPTQ calibration (8 ranks, ~15h wall)
+bash scripts/run_phase2.sh
+# Phase 3 — postprocess (rename + config patch + FP32 restore + MTP aliases)
+bash scripts/postprocess_phase2.sh
+# Phase 4 — verify
+python scripts/verify_option_y.py /scratch/weights/w4a16-fp8-mtp-gptq
+# Phase 5 — serve (see Quick start above for serve command)
+```
+## Upstream contributions filed during this work
+| Contribution | Description | Status |
+|---|---|---|
+| transformers — `save_pretrained` silent FP32 → BF16 downcast | 417 tensors specified as FP32 in DeepSeek's release spec (HC plumbing, gate bias, attn_sink, indexer/compressor `ape`) are silently written as BF16 by `save_pretrained` when model `torch_dtype` is BF16. Workaround: postprocess restore from BF16 source via [`scripts/fixup_artifact.py`](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/blob/main/scripts/fixup_artifact.py). Upstream filing pending | local |
+| vLLM — MTP loader silently skips top-level `head.weight` + `embed.weight` | `DeepSeekV4MTP.load_weights` calls `name.replace("mtp.0.", "")` which no-ops on non-`mtp.0.*` keys; `get_spec_layer_idx` returns None → loop skips. `head.weight` and `embed.weight` never reach `shared_head.head` / `embed_tokens` → uninitialized → **0% MTP acceptance with no load-time error**. Workaround: postprocess injects `mtp.0.head.weight` and `mtp.0.emb.tok_emb.weight` as duplicates. Upstream filing pending | local |
+| vLLM — DeepGemm `paged_mqa_logits` asserts on `num_speculative_tokens > 1` | `smxx_fp8_fp4_paged_mqa_logits.hpp:233` enforces `next_n == 1 or next_n == 2`. With `next_n = k+1`, practical k cap is 1. Caps spec-decode speedup at 1.49× vs sibling's published 2.03× at k=2 | upstream (DeepGemm) — filing pending |
+| [`vllm-project/vllm#43248`](https://github.com/vllm-project/vllm/pull/43248) | `bool()` wrap on `is_static_input_scheme` | open |
+| [`vllm-project/vllm#43288`](https://github.com/vllm-project/vllm/pull/43288) | `scale_fmt` defensive `.get()` + BF16 `getattr` wrap | open |
+| [`vllm-project/vllm#43290`](https://github.com/vllm-project/vllm/pull/43290) | `weight_scale_inv`-or-`weight_scale` fallback | open |
+| [`vllm-project/vllm#43319`](https://github.com/vllm-project/vllm/pull/43319) | MTP-quant-detect from safetensors header + BF16 `wo_a` fallback path | open |
+## Changes
+| Date | Change |
+|---|---|
+| 2026-05-22 | Initial release on H200 (`jasl/vllm@ds4-sm120-experimental@abad5dc71`). GSM8K 93.71% strict, MMLU 86.88%, HumanEval 84.76%, MTP acceptance 89% on calibrated workload / 70% on random prompts |
+| 2026-05-24 (morning) | RTX PRO 6000 Blackwell (SM 12.0) added. TP=2 and TP=4 both validated, chat-smoke 4/4 PASS, MTP acceptance 68-72%, MTP-on per-replica throughput 98.83 tok/s @ TP=2 / 107.32 @ TP=4. Per-replica throughput beats H200 at every batch size. `vllm-project/vllm#41511` (Marlin TP > 2 bug) did **not** fire on this build |
+| 2026-05-24 (afternoon) | **Shipping-bug fix.** Artifact previously shipped FP8_BLOCK compressor/indexer with `.weight_scale` keys; current upstream/preview-dev vLLM constructs those modules as plain BF16 (`quant_config=None`), so the artifact failed to load with `KeyError: 'layers.10.attn.mla_attn.compressor.fused_wkv_wgate.weight_scale'`. 166 compressor/indexer weights dequantized in-place (FP8 + BF16 scale → BF16, mathematically lossless) and re-uploaded. Artifact now loads cleanly on modern vLLM on both RTX PRO 6000 and H200 without local preprocessing. H200 historical numbers above remain valid: the original H200 build supported FP8 compressor; modern vLLM serving the new BF16-compressor format produces equivalent outputs (verified post-fix on RTX PRO 6000 TP=4: GSM8K-50 chat-mode 44/50 = 88.0%, matches sibling Card B's 88% strict TP=2 / 90% strict TP=4 on this hardware). |
+| 2026-05-24 (afternoon) | **AIME 2024 methodology correction.** Prior 30.0% `exact_match` was an lm-eval-harness `aime24` task artifact (completions-mode prompt, no chat template, exact-string scorer on a thinking model whose answers are `\boxed{N}`). 1-shot smoke under chat-templated thinking=high returns the correct integer. Full re-bench blocked by the RTX PRO 6000 concurrent-thinking CUDA crash (see Honest limitations); deferred to H200. Prior 30.0% struck through in Quality table. |
+| 2026-05-24 (evening) | **Root-cause investigation of the RTX PRO 6000 concurrent-thinking issue.** Updated to `jasl/vllm@a937d4b28` (Stabilize SM12x sparse MLA long prefill) — server no longer crashes under concurrent thinking-mode load, but produces **token-stream corruption** on ~50% of long generations at c=4. Tested workarounds (`VLLM_TRITON_MLA_SPARSE_TOPK_CHUNK_SIZE=256`, `VLLM_TRITON_MLA_SPARSE_MATMUL_DECODE=0`, MTP-off, eager-mode, concurrency sweep) — none reduce corruption; all increase crash rate. **Diagnostic comparison:** same hardware + same build + same workload on Card B (NVFP4 + flashinfer_trtllm MoE) = **1/30 corrupted** vs Card D (W4A16 + Marlin MoE) = **14/30 corrupted**. Bug isolated to **W4A16 + Marlin MoE decode path on SM 12.0**. Production-config verified on Card D: GSM8K-20 chat-mode sequential = 20/20 = 100%, MTP draft acceptance 92.46%. Full debug log: [`docs/findings/sm12x_token_corruption_2026_05_24.md`](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/blob/main/docs/findings/sm12x_token_corruption_2026_05_24.md). Filed upstream: [`jasl/vllm#12`](https://github.com/jasl/vllm/issues/12). |
+| 2026-05-25 | **Applied [`vllm-project/vllm#40923`](https://github.com/vllm-project/vllm/pull/40923) "Marlin MoE: include SM 12.x in default arch list" + clean rebuild.** PR #40923's description matches our symptom verbatim ("V4-Flash MoE decode emits gibberish on RTX 50-series GB10/DGX Spark... driver JIT-promotes 8.0+PTX fallback"). After applying the patch (12.0a;12.1a for CUDA 12.9) and forcing a full Marlin MoE source regen + rebuild: **AIME c=4 thinking corruption dropped from 14/30 → 0/30**, but the underlying kernel race surfaced as a different failure mode — `CUDA error: an illegal memory access was encountered` in Worker_TP*, with 29/30 errors and 1/30 completing correctly. **PR #40923 is necessary but not sufficient**: native SM 12.0a Marlin MoE cubins eliminate the JIT-PTX corruption, but a second race in the W4A16 Marlin MoE decode path under concurrent thinking-mode on SM 12.0 still crashes the worker. NVFP4 sibling [`canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP) remains the recommendation for batched thinking-mode on this hardware. 1-shot smoke + sequential workloads still clean. PR #40923 status: OPEN, member-approved 2026-04-27 by Harry-Chen, blocked on core-maintainer SM120 policy review; canada-quant repro will be posted as additional evidence. |
+## Files in the artifact
+- 4 sharded `model-*.safetensors` files + `model.safetensors.index.json` (159 GB total)
+- `config.json` — vLLM-compatible quantization_config with MTP block excluded
+- `tokenizer.json`, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja` — upstream DSV4-Flash
+- `recipe.yaml` — the llm-compressor GPTQ recipe
+- `README.md` — this file
+## Citation
+```bibtex
+@misc{canada-quant-dsv4-flash-w4a16-fp8-mtp-2026,
+  title  = {DeepSeek-V4-Flash W4A16-FP8 with BF16 MTP retained for vLLM speculative decoding},
+  author = {Canada Quant},
+  year   = {2026},
+  publisher = {Hugging Face},
+  url    = {https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP}
+}
+```
+## License
+MIT, inherited from upstream `deepseek-ai/DeepSeek-V4-Flash`. Review at the upstream repo before commercial deployment.
+## Acknowledgments
+- **DeepSeek** for the base model + MTP architecture + inference reference.
+- **jasl** ([`jasl/vllm`](https://github.com/jasl/vllm) and [`jasl/vllm-ds4-sm120-harness`](https://github.com/jasl/vllm-ds4-sm120-harness)) for the vLLM build pins (`ds4-sm120-experimental` for H200; `ds4-sm120-preview-dev` for RTX PRO 6000 SM 12.0) and the benchmark harness.
+- **[`canada-quant/DeepSeek-V4-Flash-W4A16-FP8`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8)** (predecessor) for the proven recipe topology this artifact extends with MTP.
+- **[`canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP)** (sibling) for the alias-injection pattern and MTP acceptance methodology.
+- vLLM, llm-compressor, compressed-tensors, FlashInfer maintainers.

config.json ADDED Viewed

	@@ -0,0 +1,1163 @@

+{
+  "architectures": [
+    "DeepseekV4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "compress_rates": {
+    "compressed_sparse_attention": 4,
+    "heavily_compressed_attention": 128
+  },
+  "compress_rope_theta": 160000,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "expert_dtype": "bf16",
+  "hc_eps": 1e-06,
+  "hc_mult": 4,
+  "hc_sinkhorn_iters": 20,
+  "head_dim": 512,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "index_head_dim": 128,
+  "index_n_heads": 64,
+  "index_topk": 512,
+  "initializer_range": 0.02,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention",
+    "heavily_compressed_attention",
+    "compressed_sparse_attention"
+  ],
+  "max_position_embeddings": 1048576,
+  "mlp_bias": false,
+  "mlp_layer_types": [
+    "hash_moe",
+    "hash_moe",
+    "hash_moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe",
+    "moe"
+  ],
+  "model_type": "deepseek_v4",
+  "moe_intermediate_size": 2048,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 43,
+  "num_key_value_heads": 1,
+  "num_nextn_predict_layers": 1,
+  "o_groups": 8,
+  "o_lora_rank": 1024,
+  "output_router_logits": false,
+  "pad_token_id": null,
+  "partial_rotary_factor": 0.125,
+  "q_lora_rank": 1024,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": 128,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "group",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        },
+        "output_activations": null,
+        "targets": [
+          "re:.*attn\\.(wq_a|wq_b|wkv|wo_a|wo_b|fused_wqa_wkv|q_a_proj|q_b_proj|kv_proj|o_a_proj|o_b_proj)$"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": [
+            128,
+            128
+          ],
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "block",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        }
+      },
+      "group_1": {
+        "format": "pack-quantized",
+        "input_activations": null,
+        "output_activations": null,
+        "targets": [
+          "re:.*experts\\.\\d+\\.(w1|w2|w3|gate_proj|up_proj|down_proj|gate_up_proj)$"
+        ],
+        "weights": {
+          "actorder": "static",
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": 128,
+          "num_bits": 4,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "group",
+          "symmetric": true,
+          "type": "int",
+          "zp_dtype": null
+        }
+      }
+    },
+    "format": "mixed-precision",
+    "global_compression_ratio": null,
+    "ignore": [
+      "layers.0.ffn.shared_experts.w1",
+      "layers.0.ffn.shared_experts.w2",
+      "layers.0.ffn.shared_experts.w3",
+      "layers.1.ffn.shared_experts.w1",
+      "layers.1.ffn.shared_experts.w2",
+      "layers.1.ffn.shared_experts.w3",
+      "layers.2.ffn.shared_experts.w1",
+      "layers.2.ffn.shared_experts.w2",
+      "layers.2.ffn.shared_experts.w3",
+      "layers.3.ffn.shared_experts.w1",
+      "layers.3.ffn.shared_experts.w2",
+      "layers.3.ffn.shared_experts.w3",
+      "layers.4.ffn.shared_experts.w1",
+      "layers.4.ffn.shared_experts.w2",
+      "layers.4.ffn.shared_experts.w3",
+      "layers.5.ffn.shared_experts.w1",
+      "layers.5.ffn.shared_experts.w2",
+      "layers.5.ffn.shared_experts.w3",
+      "layers.6.ffn.shared_experts.w1",
+      "layers.6.ffn.shared_experts.w2",
+      "layers.6.ffn.shared_experts.w3",
+      "layers.7.ffn.shared_experts.w1",
+      "layers.7.ffn.shared_experts.w2",
+      "layers.7.ffn.shared_experts.w3",
+      "layers.8.ffn.shared_experts.w1",
+      "layers.8.ffn.shared_experts.w2",
+      "layers.8.ffn.shared_experts.w3",
+      "layers.9.ffn.shared_experts.w1",
+      "layers.9.ffn.shared_experts.w2",
+      "layers.9.ffn.shared_experts.w3",
+      "layers.10.ffn.shared_experts.w1",
+      "layers.10.ffn.shared_experts.w2",
+      "layers.10.ffn.shared_experts.w3",
+      "layers.11.ffn.shared_experts.w1",
+      "layers.11.ffn.shared_experts.w2",
+      "layers.11.ffn.shared_experts.w3",
+      "layers.12.ffn.shared_experts.w1",
+      "layers.12.ffn.shared_experts.w2",
+      "layers.12.ffn.shared_experts.w3",
+      "layers.13.ffn.shared_experts.w1",
+      "layers.13.ffn.shared_experts.w2",
+      "layers.13.ffn.shared_experts.w3",
+      "layers.14.ffn.shared_experts.w1",
+      "layers.14.ffn.shared_experts.w2",
+      "layers.14.ffn.shared_experts.w3",
+      "layers.15.ffn.shared_experts.w1",
+      "layers.15.ffn.shared_experts.w2",
+      "layers.15.ffn.shared_experts.w3",
+      "layers.16.ffn.shared_experts.w1",
+      "layers.16.ffn.shared_experts.w2",
+      "layers.16.ffn.shared_experts.w3",
+      "layers.17.ffn.shared_experts.w1",
+      "layers.17.ffn.shared_experts.w2",
+      "layers.17.ffn.shared_experts.w3",
+      "layers.18.ffn.shared_experts.w1",
+      "layers.18.ffn.shared_experts.w2",
+      "layers.18.ffn.shared_experts.w3",
+      "layers.19.ffn.shared_experts.w1",
+      "layers.19.ffn.shared_experts.w2",
+      "layers.19.ffn.shared_experts.w3",
+      "layers.20.ffn.shared_experts.w1",
+      "layers.20.ffn.shared_experts.w2",
+      "layers.20.ffn.shared_experts.w3",
+      "layers.21.ffn.shared_experts.w1",
+      "layers.21.ffn.shared_experts.w2",
+      "layers.21.ffn.shared_experts.w3",
+      "layers.22.ffn.shared_experts.w1",
+      "layers.22.ffn.shared_experts.w2",
+      "layers.22.ffn.shared_experts.w3",
+      "layers.23.ffn.shared_experts.w1",
+      "layers.23.ffn.shared_experts.w2",
+      "layers.23.ffn.shared_experts.w3",
+      "layers.24.ffn.shared_experts.w1",
+      "layers.24.ffn.shared_experts.w2",
+      "layers.24.ffn.shared_experts.w3",
+      "layers.25.ffn.shared_experts.w1",
+      "layers.25.ffn.shared_experts.w2",
+      "layers.25.ffn.shared_experts.w3",
+      "layers.26.ffn.shared_experts.w1",
+      "layers.26.ffn.shared_experts.w2",
+      "layers.26.ffn.shared_experts.w3",
+      "layers.27.ffn.shared_experts.w1",
+      "layers.27.ffn.shared_experts.w2",
+      "layers.27.ffn.shared_experts.w3",
+      "layers.28.ffn.shared_experts.w1",
+      "layers.28.ffn.shared_experts.w2",
+      "layers.28.ffn.shared_experts.w3",
+      "layers.29.ffn.shared_experts.w1",
+      "layers.29.ffn.shared_experts.w2",
+      "layers.29.ffn.shared_experts.w3",
+      "layers.30.ffn.shared_experts.w1",
+      "layers.30.ffn.shared_experts.w2",
+      "layers.30.ffn.shared_experts.w3",
+      "layers.31.ffn.shared_experts.w1",
+      "layers.31.ffn.shared_experts.w2",
+      "layers.31.ffn.shared_experts.w3",
+      "layers.32.ffn.shared_experts.w1",
+      "layers.32.ffn.shared_experts.w2",
+      "layers.32.ffn.shared_experts.w3",
+      "layers.33.ffn.shared_experts.w1",
+      "layers.33.ffn.shared_experts.w2",
+      "layers.33.ffn.shared_experts.w3",
+      "layers.34.ffn.shared_experts.w1",
+      "layers.34.ffn.shared_experts.w2",
+      "layers.34.ffn.shared_experts.w3",
+      "layers.35.ffn.shared_experts.w1",
+      "layers.35.ffn.shared_experts.w2",
+      "layers.35.ffn.shared_experts.w3",
+      "layers.36.ffn.shared_experts.w1",
+      "layers.36.ffn.shared_experts.w2",
+      "layers.36.ffn.shared_experts.w3",
+      "layers.37.ffn.shared_experts.w1",
+      "layers.37.ffn.shared_experts.w2",
+      "layers.37.ffn.shared_experts.w3",
+      "layers.38.ffn.shared_experts.w1",
+      "layers.38.ffn.shared_experts.w2",
+      "layers.38.ffn.shared_experts.w3",
+      "layers.39.ffn.shared_experts.w1",
+      "layers.39.ffn.shared_experts.w2",
+      "layers.39.ffn.shared_experts.w3",
+      "layers.40.ffn.shared_experts.w1",
+      "layers.40.ffn.shared_experts.w2",
+      "layers.40.ffn.shared_experts.w3",
+      "layers.41.ffn.shared_experts.w1",
+      "layers.41.ffn.shared_experts.w2",
+      "layers.41.ffn.shared_experts.w3",
+      "layers.42.ffn.shared_experts.w1",
+      "layers.42.ffn.shared_experts.w2",
+      "layers.42.ffn.shared_experts.w3",
+      "re:^layers\\.43\\.",
+      "re:^model\\.layers\\.43\\.",
+      "lm_head",
+      "layers.0.attn.compressor.wgate",
+      "layers.0.attn.compressor.wkv",
+      "layers.0.attn.compressor.fused_wkv_wgate",
+      "layers.0.attn.compressor.gate_proj",
+      "layers.0.attn.compressor.kv_proj",
+      "layers.0.attn.indexer.weights_proj",
+      "layers.0.attn.indexer.wq_b",
+      "layers.0.attn.indexer.q_b_proj",
+      "layers.0.attn.indexer.compressor.wgate",
+      "layers.0.attn.indexer.compressor.wkv",
+      "layers.0.attn.indexer.compressor.gate_proj",
+      "layers.0.attn.indexer.compressor.kv_proj",
+      "layers.1.attn.compressor.wgate",
+      "layers.1.attn.compressor.wkv",
+      "layers.1.attn.compressor.fused_wkv_wgate",
+      "layers.1.attn.compressor.gate_proj",
+      "layers.1.attn.compressor.kv_proj",
+      "layers.1.attn.indexer.weights_proj",
+      "layers.1.attn.indexer.wq_b",
+      "layers.1.attn.indexer.q_b_proj",
+      "layers.1.attn.indexer.compressor.wgate",
+      "layers.1.attn.indexer.compressor.wkv",
+      "layers.1.attn.indexer.compressor.gate_proj",
+      "layers.1.attn.indexer.compressor.kv_proj",
+      "layers.2.attn.compressor.wgate",
+      "layers.2.attn.compressor.wkv",
+      "layers.2.attn.compressor.fused_wkv_wgate",
+      "layers.2.attn.compressor.gate_proj",
+      "layers.2.attn.compressor.kv_proj",
+      "layers.2.attn.indexer.weights_proj",
+      "layers.2.attn.indexer.wq_b",
+      "layers.2.attn.indexer.q_b_proj",
+      "layers.2.attn.indexer.compressor.wgate",
+      "layers.2.attn.indexer.compressor.wkv",
+      "layers.2.attn.indexer.compressor.gate_proj",
+      "layers.2.attn.indexer.compressor.kv_proj",
+      "layers.3.attn.compressor.wgate",
+      "layers.3.attn.compressor.wkv",
+      "layers.3.attn.compressor.fused_wkv_wgate",
+      "layers.3.attn.compressor.gate_proj",
+      "layers.3.attn.compressor.kv_proj",
+      "layers.3.attn.indexer.weights_proj",
+      "layers.3.attn.indexer.wq_b",
+      "layers.3.attn.indexer.q_b_proj",
+      "layers.3.attn.indexer.compressor.wgate",
+      "layers.3.attn.indexer.compressor.wkv",
+      "layers.3.attn.indexer.compressor.gate_proj",
+      "layers.3.attn.indexer.compressor.kv_proj",
+      "layers.4.attn.compressor.wgate",
+      "layers.4.attn.compressor.wkv",
+      "layers.4.attn.compressor.fused_wkv_wgate",
+      "layers.4.attn.compressor.gate_proj",
+      "layers.4.attn.compressor.kv_proj",
+      "layers.4.attn.indexer.weights_proj",
+      "layers.4.attn.indexer.wq_b",
+      "layers.4.attn.indexer.q_b_proj",
+      "layers.4.attn.indexer.compressor.wgate",
+      "layers.4.attn.indexer.compressor.wkv",
+      "layers.4.attn.indexer.compressor.gate_proj",
+      "layers.4.attn.indexer.compressor.kv_proj",
+      "layers.5.attn.compressor.wgate",
+      "layers.5.attn.compressor.wkv",
+      "layers.5.attn.compressor.fused_wkv_wgate",
+      "layers.5.attn.compressor.gate_proj",
+      "layers.5.attn.compressor.kv_proj",
+      "layers.5.attn.indexer.weights_proj",
+      "layers.5.attn.indexer.wq_b",
+      "layers.5.attn.indexer.q_b_proj",
+      "layers.5.attn.indexer.compressor.wgate",
+      "layers.5.attn.indexer.compressor.wkv",
+      "layers.5.attn.indexer.compressor.gate_proj",
+      "layers.5.attn.indexer.compressor.kv_proj",
+      "layers.6.attn.compressor.wgate",
+      "layers.6.attn.compressor.wkv",
+      "layers.6.attn.compressor.fused_wkv_wgate",
+      "layers.6.attn.compressor.gate_proj",
+      "layers.6.attn.compressor.kv_proj",
+      "layers.6.attn.indexer.weights_proj",
+      "layers.6.attn.indexer.wq_b",
+      "layers.6.attn.indexer.q_b_proj",
+      "layers.6.attn.indexer.compressor.wgate",
+      "layers.6.attn.indexer.compressor.wkv",
+      "layers.6.attn.indexer.compressor.gate_proj",
+      "layers.6.attn.indexer.compressor.kv_proj",
+      "layers.7.attn.compressor.wgate",
+      "layers.7.attn.compressor.wkv",
+      "layers.7.attn.compressor.fused_wkv_wgate",
+      "layers.7.attn.compressor.gate_proj",
+      "layers.7.attn.compressor.kv_proj",
+      "layers.7.attn.indexer.weights_proj",
+      "layers.7.attn.indexer.wq_b",
+      "layers.7.attn.indexer.q_b_proj",
+      "layers.7.attn.indexer.compressor.wgate",
+      "layers.7.attn.indexer.compressor.wkv",
+      "layers.7.attn.indexer.compressor.gate_proj",
+      "layers.7.attn.indexer.compressor.kv_proj",
+      "layers.8.attn.compressor.wgate",
+      "layers.8.attn.compressor.wkv",
+      "layers.8.attn.compressor.fused_wkv_wgate",
+      "layers.8.attn.compressor.gate_proj",
+      "layers.8.attn.compressor.kv_proj",
+      "layers.8.attn.indexer.weights_proj",
+      "layers.8.attn.indexer.wq_b",
+      "layers.8.attn.indexer.q_b_proj",
+      "layers.8.attn.indexer.compressor.wgate",
+      "layers.8.attn.indexer.compressor.wkv",
+      "layers.8.attn.indexer.compressor.gate_proj",
+      "layers.8.attn.indexer.compressor.kv_proj",
+      "layers.9.attn.compressor.wgate",
+      "layers.9.attn.compressor.wkv",
+      "layers.9.attn.compressor.fused_wkv_wgate",
+      "layers.9.attn.compressor.gate_proj",
+      "layers.9.attn.compressor.kv_proj",
+      "layers.9.attn.indexer.weights_proj",
+      "layers.9.attn.indexer.wq_b",
+      "layers.9.attn.indexer.q_b_proj",
+      "layers.9.attn.indexer.compressor.wgate",
+      "layers.9.attn.indexer.compressor.wkv",
+      "layers.9.attn.indexer.compressor.gate_proj",
+      "layers.9.attn.indexer.compressor.kv_proj",
+      "layers.10.attn.compressor.wgate",
+      "layers.10.attn.compressor.wkv",
+      "layers.10.attn.compressor.fused_wkv_wgate",
+      "layers.10.attn.compressor.gate_proj",
+      "layers.10.attn.compressor.kv_proj",
+      "layers.10.attn.indexer.weights_proj",
+      "layers.10.attn.indexer.wq_b",
+      "layers.10.attn.indexer.q_b_proj",
+      "layers.10.attn.indexer.compressor.wgate",
+      "layers.10.attn.indexer.compressor.wkv",
+      "layers.10.attn.indexer.compressor.gate_proj",
+      "layers.10.attn.indexer.compressor.kv_proj",
+      "layers.11.attn.compressor.wgate",
+      "layers.11.attn.compressor.wkv",
+      "layers.11.attn.compressor.fused_wkv_wgate",
+      "layers.11.attn.compressor.gate_proj",
+      "layers.11.attn.compressor.kv_proj",
+      "layers.11.attn.indexer.weights_proj",
+      "layers.11.attn.indexer.wq_b",
+      "layers.11.attn.indexer.q_b_proj",
+      "layers.11.attn.indexer.compressor.wgate",
+      "layers.11.attn.indexer.compressor.wkv",
+      "layers.11.attn.indexer.compressor.gate_proj",
+      "layers.11.attn.indexer.compressor.kv_proj",
+      "layers.12.attn.compressor.wgate",
+      "layers.12.attn.compressor.wkv",
+      "layers.12.attn.compressor.fused_wkv_wgate",
+      "layers.12.attn.compressor.gate_proj",
+      "layers.12.attn.compressor.kv_proj",
+      "layers.12.attn.indexer.weights_proj",
+      "layers.12.attn.indexer.wq_b",
+      "layers.12.attn.indexer.q_b_proj",
+      "layers.12.attn.indexer.compressor.wgate",
+      "layers.12.attn.indexer.compressor.wkv",
+      "layers.12.attn.indexer.compressor.gate_proj",
+      "layers.12.attn.indexer.compressor.kv_proj",
+      "layers.13.attn.compressor.wgate",
+      "layers.13.attn.compressor.wkv",
+      "layers.13.attn.compressor.fused_wkv_wgate",
+      "layers.13.attn.compressor.gate_proj",
+      "layers.13.attn.compressor.kv_proj",
+      "layers.13.attn.indexer.weights_proj",
+      "layers.13.attn.indexer.wq_b",
+      "layers.13.attn.indexer.q_b_proj",
+      "layers.13.attn.indexer.compressor.wgate",
+      "layers.13.attn.indexer.compressor.wkv",
+      "layers.13.attn.indexer.compressor.gate_proj",
+      "layers.13.attn.indexer.compressor.kv_proj",
+      "layers.14.attn.compressor.wgate",
+      "layers.14.attn.compressor.wkv",
+      "layers.14.attn.compressor.fused_wkv_wgate",
+      "layers.14.attn.compressor.gate_proj",
+      "layers.14.attn.compressor.kv_proj",
+      "layers.14.attn.indexer.weights_proj",
+      "layers.14.attn.indexer.wq_b",
+      "layers.14.attn.indexer.q_b_proj",
+      "layers.14.attn.indexer.compressor.wgate",
+      "layers.14.attn.indexer.compressor.wkv",
+      "layers.14.attn.indexer.compressor.gate_proj",
+      "layers.14.attn.indexer.compressor.kv_proj",
+      "layers.15.attn.compressor.wgate",
+      "layers.15.attn.compressor.wkv",
+      "layers.15.attn.compressor.fused_wkv_wgate",
+      "layers.15.attn.compressor.gate_proj",
+      "layers.15.attn.compressor.kv_proj",
+      "layers.15.attn.indexer.weights_proj",
+      "layers.15.attn.indexer.wq_b",
+      "layers.15.attn.indexer.q_b_proj",
+      "layers.15.attn.indexer.compressor.wgate",
+      "layers.15.attn.indexer.compressor.wkv",
+      "layers.15.attn.indexer.compressor.gate_proj",
+      "layers.15.attn.indexer.compressor.kv_proj",
+      "layers.16.attn.compressor.wgate",
+      "layers.16.attn.compressor.wkv",
+      "layers.16.attn.compressor.fused_wkv_wgate",
+      "layers.16.attn.compressor.gate_proj",
+      "layers.16.attn.compressor.kv_proj",
+      "layers.16.attn.indexer.weights_proj",
+      "layers.16.attn.indexer.wq_b",
+      "layers.16.attn.indexer.q_b_proj",
+      "layers.16.attn.indexer.compressor.wgate",
+      "layers.16.attn.indexer.compressor.wkv",
+      "layers.16.attn.indexer.compressor.gate_proj",
+      "layers.16.attn.indexer.compressor.kv_proj",
+      "layers.17.attn.compressor.wgate",
+      "layers.17.attn.compressor.wkv",
+      "layers.17.attn.compressor.fused_wkv_wgate",
+      "layers.17.attn.compressor.gate_proj",
+      "layers.17.attn.compressor.kv_proj",
+      "layers.17.attn.indexer.weights_proj",
+      "layers.17.attn.indexer.wq_b",
+      "layers.17.attn.indexer.q_b_proj",
+      "layers.17.attn.indexer.compressor.wgate",
+      "layers.17.attn.indexer.compressor.wkv",
+      "layers.17.attn.indexer.compressor.gate_proj",
+      "layers.17.attn.indexer.compressor.kv_proj",
+      "layers.18.attn.compressor.wgate",
+      "layers.18.attn.compressor.wkv",
+      "layers.18.attn.compressor.fused_wkv_wgate",
+      "layers.18.attn.compressor.gate_proj",
+      "layers.18.attn.compressor.kv_proj",
+      "layers.18.attn.indexer.weights_proj",
+      "layers.18.attn.indexer.wq_b",
+      "layers.18.attn.indexer.q_b_proj",
+      "layers.18.attn.indexer.compressor.wgate",
+      "layers.18.attn.indexer.compressor.wkv",
+      "layers.18.attn.indexer.compressor.gate_proj",
+      "layers.18.attn.indexer.compressor.kv_proj",
+      "layers.19.attn.compressor.wgate",
+      "layers.19.attn.compressor.wkv",
+      "layers.19.attn.compressor.fused_wkv_wgate",
+      "layers.19.attn.compressor.gate_proj",
+      "layers.19.attn.compressor.kv_proj",
+      "layers.19.attn.indexer.weights_proj",
+      "layers.19.attn.indexer.wq_b",
+      "layers.19.attn.indexer.q_b_proj",
+      "layers.19.attn.indexer.compressor.wgate",
+      "layers.19.attn.indexer.compressor.wkv",
+      "layers.19.attn.indexer.compressor.gate_proj",
+      "layers.19.attn.indexer.compressor.kv_proj",
+      "layers.20.attn.compressor.wgate",
+      "layers.20.attn.compressor.wkv",
+      "layers.20.attn.compressor.fused_wkv_wgate",
+      "layers.20.attn.compressor.gate_proj",
+      "layers.20.attn.compressor.kv_proj",
+      "layers.20.attn.indexer.weights_proj",
+      "layers.20.attn.indexer.wq_b",
+      "layers.20.attn.indexer.q_b_proj",
+      "layers.20.attn.indexer.compressor.wgate",
+      "layers.20.attn.indexer.compressor.wkv",
+      "layers.20.attn.indexer.compressor.gate_proj",
+      "layers.20.attn.indexer.compressor.kv_proj",
+      "layers.21.attn.compressor.wgate",
+      "layers.21.attn.compressor.wkv",
+      "layers.21.attn.compressor.fused_wkv_wgate",
+      "layers.21.attn.compressor.gate_proj",
+      "layers.21.attn.compressor.kv_proj",
+      "layers.21.attn.indexer.weights_proj",
+      "layers.21.attn.indexer.wq_b",
+      "layers.21.attn.indexer.q_b_proj",
+      "layers.21.attn.indexer.compressor.wgate",
+      "layers.21.attn.indexer.compressor.wkv",
+      "layers.21.attn.indexer.compressor.gate_proj",
+      "layers.21.attn.indexer.compressor.kv_proj",
+      "layers.22.attn.compressor.wgate",
+      "layers.22.attn.compressor.wkv",
+      "layers.22.attn.compressor.fused_wkv_wgate",
+      "layers.22.attn.compressor.gate_proj",
+      "layers.22.attn.compressor.kv_proj",
+      "layers.22.attn.indexer.weights_proj",
+      "layers.22.attn.indexer.wq_b",
+      "layers.22.attn.indexer.q_b_proj",
+      "layers.22.attn.indexer.compressor.wgate",
+      "layers.22.attn.indexer.compressor.wkv",
+      "layers.22.attn.indexer.compressor.gate_proj",
+      "layers.22.attn.indexer.compressor.kv_proj",
+      "layers.23.attn.compressor.wgate",
+      "layers.23.attn.compressor.wkv",
+      "layers.23.attn.compressor.fused_wkv_wgate",
+      "layers.23.attn.compressor.gate_proj",
+      "layers.23.attn.compressor.kv_proj",
+      "layers.23.attn.indexer.weights_proj",
+      "layers.23.attn.indexer.wq_b",
+      "layers.23.attn.indexer.q_b_proj",
+      "layers.23.attn.indexer.compressor.wgate",
+      "layers.23.attn.indexer.compressor.wkv",
+      "layers.23.attn.indexer.compressor.gate_proj",
+      "layers.23.attn.indexer.compressor.kv_proj",
+      "layers.24.attn.compressor.wgate",
+      "layers.24.attn.compressor.wkv",
+      "layers.24.attn.compressor.fused_wkv_wgate",
+      "layers.24.attn.compressor.gate_proj",
+      "layers.24.attn.compressor.kv_proj",
+      "layers.24.attn.indexer.weights_proj",
+      "layers.24.attn.indexer.wq_b",
+      "layers.24.attn.indexer.q_b_proj",
+      "layers.24.attn.indexer.compressor.wgate",
+      "layers.24.attn.indexer.compressor.wkv",
+      "layers.24.attn.indexer.compressor.gate_proj",
+      "layers.24.attn.indexer.compressor.kv_proj",
+      "layers.25.attn.compressor.wgate",
+      "layers.25.attn.compressor.wkv",
+      "layers.25.attn.compressor.fused_wkv_wgate",
+      "layers.25.attn.compressor.gate_proj",
+      "layers.25.attn.compressor.kv_proj",
+      "layers.25.attn.indexer.weights_proj",
+      "layers.25.attn.indexer.wq_b",
+      "layers.25.attn.indexer.q_b_proj",
+      "layers.25.attn.indexer.compressor.wgate",
+      "layers.25.attn.indexer.compressor.wkv",
+      "layers.25.attn.indexer.compressor.gate_proj",
+      "layers.25.attn.indexer.compressor.kv_proj",
+      "layers.26.attn.compressor.wgate",
+      "layers.26.attn.compressor.wkv",
+      "layers.26.attn.compressor.fused_wkv_wgate",
+      "layers.26.attn.compressor.gate_proj",
+      "layers.26.attn.compressor.kv_proj",
+      "layers.26.attn.indexer.weights_proj",
+      "layers.26.attn.indexer.wq_b",
+      "layers.26.attn.indexer.q_b_proj",
+      "layers.26.attn.indexer.compressor.wgate",
+      "layers.26.attn.indexer.compressor.wkv",
+      "layers.26.attn.indexer.compressor.gate_proj",
+      "layers.26.attn.indexer.compressor.kv_proj",
+      "layers.27.attn.compressor.wgate",
+      "layers.27.attn.compressor.wkv",
+      "layers.27.attn.compressor.fused_wkv_wgate",
+      "layers.27.attn.compressor.gate_proj",
+      "layers.27.attn.compressor.kv_proj",
+      "layers.27.attn.indexer.weights_proj",
+      "layers.27.attn.indexer.wq_b",
+      "layers.27.attn.indexer.q_b_proj",
+      "layers.27.attn.indexer.compressor.wgate",
+      "layers.27.attn.indexer.compressor.wkv",
+      "layers.27.attn.indexer.compressor.gate_proj",
+      "layers.27.attn.indexer.compressor.kv_proj",
+      "layers.28.attn.compressor.wgate",
+      "layers.28.attn.compressor.wkv",
+      "layers.28.attn.compressor.fused_wkv_wgate",
+      "layers.28.attn.compressor.gate_proj",
+      "layers.28.attn.compressor.kv_proj",
+      "layers.28.attn.indexer.weights_proj",
+      "layers.28.attn.indexer.wq_b",
+      "layers.28.attn.indexer.q_b_proj",
+      "layers.28.attn.indexer.compressor.wgate",
+      "layers.28.attn.indexer.compressor.wkv",
+      "layers.28.attn.indexer.compressor.gate_proj",
+      "layers.28.attn.indexer.compressor.kv_proj",
+      "layers.29.attn.compressor.wgate",
+      "layers.29.attn.compressor.wkv",
+      "layers.29.attn.compressor.fused_wkv_wgate",
+      "layers.29.attn.compressor.gate_proj",
+      "layers.29.attn.compressor.kv_proj",
+      "layers.29.attn.indexer.weights_proj",
+      "layers.29.attn.indexer.wq_b",
+      "layers.29.attn.indexer.q_b_proj",
+      "layers.29.attn.indexer.compressor.wgate",
+      "layers.29.attn.indexer.compressor.wkv",
+      "layers.29.attn.indexer.compressor.gate_proj",
+      "layers.29.attn.indexer.compressor.kv_proj",
+      "layers.30.attn.compressor.wgate",
+      "layers.30.attn.compressor.wkv",
+      "layers.30.attn.compressor.fused_wkv_wgate",
+      "layers.30.attn.compressor.gate_proj",
+      "layers.30.attn.compressor.kv_proj",
+      "layers.30.attn.indexer.weights_proj",
+      "layers.30.attn.indexer.wq_b",
+      "layers.30.attn.indexer.q_b_proj",
+      "layers.30.attn.indexer.compressor.wgate",
+      "layers.30.attn.indexer.compressor.wkv",
+      "layers.30.attn.indexer.compressor.gate_proj",
+      "layers.30.attn.indexer.compressor.kv_proj",
+      "layers.31.attn.compressor.wgate",
+      "layers.31.attn.compressor.wkv",
+      "layers.31.attn.compressor.fused_wkv_wgate",
+      "layers.31.attn.compressor.gate_proj",
+      "layers.31.attn.compressor.kv_proj",
+      "layers.31.attn.indexer.weights_proj",
+      "layers.31.attn.indexer.wq_b",
+      "layers.31.attn.indexer.q_b_proj",
+      "layers.31.attn.indexer.compressor.wgate",
+      "layers.31.attn.indexer.compressor.wkv",
+      "layers.31.attn.indexer.compressor.gate_proj",
+      "layers.31.attn.indexer.compressor.kv_proj",
+      "layers.32.attn.compressor.wgate",
+      "layers.32.attn.compressor.wkv",
+      "layers.32.attn.compressor.fused_wkv_wgate",
+      "layers.32.attn.compressor.gate_proj",
+      "layers.32.attn.compressor.kv_proj",
+      "layers.32.attn.indexer.weights_proj",
+      "layers.32.attn.indexer.wq_b",
+      "layers.32.attn.indexer.q_b_proj",
+      "layers.32.attn.indexer.compressor.wgate",
+      "layers.32.attn.indexer.compressor.wkv",
+      "layers.32.attn.indexer.compressor.gate_proj",
+      "layers.32.attn.indexer.compressor.kv_proj",
+      "layers.33.attn.compressor.wgate",
+      "layers.33.attn.compressor.wkv",
+      "layers.33.attn.compressor.fused_wkv_wgate",
+      "layers.33.attn.compressor.gate_proj",
+      "layers.33.attn.compressor.kv_proj",
+      "layers.33.attn.indexer.weights_proj",
+      "layers.33.attn.indexer.wq_b",
+      "layers.33.attn.indexer.q_b_proj",
+      "layers.33.attn.indexer.compressor.wgate",
+      "layers.33.attn.indexer.compressor.wkv",
+      "layers.33.attn.indexer.compressor.gate_proj",
+      "layers.33.attn.indexer.compressor.kv_proj",
+      "layers.34.attn.compressor.wgate",
+      "layers.34.attn.compressor.wkv",
+      "layers.34.attn.compressor.fused_wkv_wgate",
+      "layers.34.attn.compressor.gate_proj",
+      "layers.34.attn.compressor.kv_proj",
+      "layers.34.attn.indexer.weights_proj",
+      "layers.34.attn.indexer.wq_b",
+      "layers.34.attn.indexer.q_b_proj",
+      "layers.34.attn.indexer.compressor.wgate",
+      "layers.34.attn.indexer.compressor.wkv",
+      "layers.34.attn.indexer.compressor.gate_proj",
+      "layers.34.attn.indexer.compressor.kv_proj",
+      "layers.35.attn.compressor.wgate",
+      "layers.35.attn.compressor.wkv",
+      "layers.35.attn.compressor.fused_wkv_wgate",
+      "layers.35.attn.compressor.gate_proj",
+      "layers.35.attn.compressor.kv_proj",
+      "layers.35.attn.indexer.weights_proj",
+      "layers.35.attn.indexer.wq_b",
+      "layers.35.attn.indexer.q_b_proj",
+      "layers.35.attn.indexer.compressor.wgate",
+      "layers.35.attn.indexer.compressor.wkv",
+      "layers.35.attn.indexer.compressor.gate_proj",
+      "layers.35.attn.indexer.compressor.kv_proj",
+      "layers.36.attn.compressor.wgate",
+      "layers.36.attn.compressor.wkv",
+      "layers.36.attn.compressor.fused_wkv_wgate",
+      "layers.36.attn.compressor.gate_proj",
+      "layers.36.attn.compressor.kv_proj",
+      "layers.36.attn.indexer.weights_proj",
+      "layers.36.attn.indexer.wq_b",
+      "layers.36.attn.indexer.q_b_proj",
+      "layers.36.attn.indexer.compressor.wgate",
+      "layers.36.attn.indexer.compressor.wkv",
+      "layers.36.attn.indexer.compressor.gate_proj",
+      "layers.36.attn.indexer.compressor.kv_proj",
+      "layers.37.attn.compressor.wgate",
+      "layers.37.attn.compressor.wkv",
+      "layers.37.attn.compressor.fused_wkv_wgate",
+      "layers.37.attn.compressor.gate_proj",
+      "layers.37.attn.compressor.kv_proj",
+      "layers.37.attn.indexer.weights_proj",
+      "layers.37.attn.indexer.wq_b",
+      "layers.37.attn.indexer.q_b_proj",
+      "layers.37.attn.indexer.compressor.wgate",
+      "layers.37.attn.indexer.compressor.wkv",
+      "layers.37.attn.indexer.compressor.gate_proj",
+      "layers.37.attn.indexer.compressor.kv_proj",
+      "layers.38.attn.compressor.wgate",
+      "layers.38.attn.compressor.wkv",
+      "layers.38.attn.compressor.fused_wkv_wgate",
+      "layers.38.attn.compressor.gate_proj",
+      "layers.38.attn.compressor.kv_proj",
+      "layers.38.attn.indexer.weights_proj",
+      "layers.38.attn.indexer.wq_b",
+      "layers.38.attn.indexer.q_b_proj",
+      "layers.38.attn.indexer.compressor.wgate",
+      "layers.38.attn.indexer.compressor.wkv",
+      "layers.38.attn.indexer.compressor.gate_proj",
+      "layers.38.attn.indexer.compressor.kv_proj",
+      "layers.39.attn.compressor.wgate",
+      "layers.39.attn.compressor.wkv",
+      "layers.39.attn.compressor.fused_wkv_wgate",
+      "layers.39.attn.compressor.gate_proj",
+      "layers.39.attn.compressor.kv_proj",
+      "layers.39.attn.indexer.weights_proj",
+      "layers.39.attn.indexer.wq_b",
+      "layers.39.attn.indexer.q_b_proj",
+      "layers.39.attn.indexer.compressor.wgate",
+      "layers.39.attn.indexer.compressor.wkv",
+      "layers.39.attn.indexer.compressor.gate_proj",
+      "layers.39.attn.indexer.compressor.kv_proj",
+      "layers.40.attn.compressor.wgate",
+      "layers.40.attn.compressor.wkv",
+      "layers.40.attn.compressor.fused_wkv_wgate",
+      "layers.40.attn.compressor.gate_proj",
+      "layers.40.attn.compressor.kv_proj",
+      "layers.40.attn.indexer.weights_proj",
+      "layers.40.attn.indexer.wq_b",
+      "layers.40.attn.indexer.q_b_proj",
+      "layers.40.attn.indexer.compressor.wgate",
+      "layers.40.attn.indexer.compressor.wkv",
+      "layers.40.attn.indexer.compressor.gate_proj",
+      "layers.40.attn.indexer.compressor.kv_proj",
+      "layers.41.attn.compressor.wgate",
+      "layers.41.attn.compressor.wkv",
+      "layers.41.attn.compressor.fused_wkv_wgate",
+      "layers.41.attn.compressor.gate_proj",
+      "layers.41.attn.compressor.kv_proj",
+      "layers.41.attn.indexer.weights_proj",
+      "layers.41.attn.indexer.wq_b",
+      "layers.41.attn.indexer.q_b_proj",
+      "layers.41.attn.indexer.compressor.wgate",
+      "layers.41.attn.indexer.compressor.wkv",
+      "layers.41.attn.indexer.compressor.gate_proj",
+      "layers.41.attn.indexer.compressor.kv_proj",
+      "layers.42.attn.compressor.wgate",
+      "layers.42.attn.compressor.wkv",
+      "layers.42.attn.compressor.fused_wkv_wgate",
+      "layers.42.attn.compressor.gate_proj",
+      "layers.42.attn.compressor.kv_proj",
+      "layers.42.attn.indexer.weights_proj",
+      "layers.42.attn.indexer.wq_b",
+      "layers.42.attn.indexer.q_b_proj",
+      "layers.42.attn.indexer.compressor.wgate",
+      "layers.42.attn.indexer.compressor.wkv",
+      "layers.42.attn.indexer.compressor.gate_proj",
+      "layers.42.attn.indexer.compressor.kv_proj",
+      "layers.43.attn.compressor.wgate",
+      "layers.43.attn.compressor.wkv",
+      "layers.43.attn.compressor.fused_wkv_wgate",
+      "layers.43.attn.compressor.gate_proj",
+      "layers.43.attn.compressor.kv_proj",
+      "layers.43.attn.indexer.weights_proj",
+      "layers.43.attn.indexer.wq_b",
+      "layers.43.attn.indexer.q_b_proj",
+      "layers.43.attn.indexer.compressor.wgate",
+      "layers.43.attn.indexer.compressor.wkv",
+      "layers.43.attn.indexer.compressor.gate_proj",
+      "layers.43.attn.indexer.compressor.kv_proj",
+      "layers.44.attn.compressor.wgate",
+      "layers.44.attn.compressor.wkv",
+      "layers.44.attn.compressor.fused_wkv_wgate",
+      "layers.44.attn.compressor.gate_proj",
+      "layers.44.attn.compressor.kv_proj",
+      "layers.44.attn.indexer.weights_proj",
+      "layers.44.attn.indexer.wq_b",
+      "layers.44.attn.indexer.q_b_proj",
+      "layers.44.attn.indexer.compressor.wgate",
+      "layers.44.attn.indexer.compressor.wkv",
+      "layers.44.attn.indexer.compressor.gate_proj",
+      "layers.44.attn.indexer.compressor.kv_proj",
+      "layers.45.attn.compressor.wgate",
+      "layers.45.attn.compressor.wkv",
+      "layers.45.attn.compressor.fused_wkv_wgate",
+      "layers.45.attn.compressor.gate_proj",
+      "layers.45.attn.compressor.kv_proj",
+      "layers.45.attn.indexer.weights_proj",
+      "layers.45.attn.indexer.wq_b",
+      "layers.45.attn.indexer.q_b_proj",
+      "layers.45.attn.indexer.compressor.wgate",
+      "layers.45.attn.indexer.compressor.wkv",
+      "layers.45.attn.indexer.compressor.gate_proj",
+      "layers.45.attn.indexer.compressor.kv_proj",
+      "layers.46.attn.compressor.wgate",
+      "layers.46.attn.compressor.wkv",
+      "layers.46.attn.compressor.fused_wkv_wgate",
+      "layers.46.attn.compressor.gate_proj",
+      "layers.46.attn.compressor.kv_proj",
+      "layers.46.attn.indexer.weights_proj",
+      "layers.46.attn.indexer.wq_b",
+      "layers.46.attn.indexer.q_b_proj",
+      "layers.46.attn.indexer.compressor.wgate",
+      "layers.46.attn.indexer.compressor.wkv",
+      "layers.46.attn.indexer.compressor.gate_proj",
+      "layers.46.attn.indexer.compressor.kv_proj",
+      "layers.47.attn.compressor.wgate",
+      "layers.47.attn.compressor.wkv",
+      "layers.47.attn.compressor.fused_wkv_wgate",
+      "layers.47.attn.compressor.gate_proj",
+      "layers.47.attn.compressor.kv_proj",
+      "layers.47.attn.indexer.weights_proj",
+      "layers.47.attn.indexer.wq_b",
+      "layers.47.attn.indexer.q_b_proj",
+      "layers.47.attn.indexer.compressor.wgate",
+      "layers.47.attn.indexer.compressor.wkv",
+      "layers.47.attn.indexer.compressor.gate_proj",
+      "layers.47.attn.indexer.compressor.kv_proj",
+      "layers.48.attn.compressor.wgate",
+      "layers.48.attn.compressor.wkv",
+      "layers.48.attn.compressor.fused_wkv_wgate",
+      "layers.48.attn.compressor.gate_proj",
+      "layers.48.attn.compressor.kv_proj",
+      "layers.48.attn.indexer.weights_proj",
+      "layers.48.attn.indexer.wq_b",
+      "layers.48.attn.indexer.q_b_proj",
+      "layers.48.attn.indexer.compressor.wgate",
+      "layers.48.attn.indexer.compressor.wkv",
+      "layers.48.attn.indexer.compressor.gate_proj",
+      "layers.48.attn.indexer.compressor.kv_proj",
+      "layers.49.attn.compressor.wgate",
+      "layers.49.attn.compressor.wkv",
+      "layers.49.attn.compressor.fused_wkv_wgate",
+      "layers.49.attn.compressor.gate_proj",
+      "layers.49.attn.compressor.kv_proj",
+      "layers.49.attn.indexer.weights_proj",
+      "layers.49.attn.indexer.wq_b",
+      "layers.49.attn.indexer.q_b_proj",
+      "layers.49.attn.indexer.compressor.wgate",
+      "layers.49.attn.indexer.compressor.wkv",
+      "layers.49.attn.indexer.compressor.gate_proj",
+      "layers.49.attn.indexer.compressor.kv_proj",
+      "layers.50.attn.compressor.wgate",
+      "layers.50.attn.compressor.wkv",
+      "layers.50.attn.compressor.fused_wkv_wgate",
+      "layers.50.attn.compressor.gate_proj",
+      "layers.50.attn.compressor.kv_proj",
+      "layers.50.attn.indexer.weights_proj",
+      "layers.50.attn.indexer.wq_b",
+      "layers.50.attn.indexer.q_b_proj",
+      "layers.50.attn.indexer.compressor.wgate",
+      "layers.50.attn.indexer.compressor.wkv",
+      "layers.50.attn.indexer.compressor.gate_proj",
+      "layers.50.attn.indexer.compressor.kv_proj",
+      "layers.51.attn.compressor.wgate",
+      "layers.51.attn.compressor.wkv",
+      "layers.51.attn.compressor.fused_wkv_wgate",
+      "layers.51.attn.compressor.gate_proj",
+      "layers.51.attn.compressor.kv_proj",
+      "layers.51.attn.indexer.weights_proj",
+      "layers.51.attn.indexer.wq_b",
+      "layers.51.attn.indexer.q_b_proj",
+      "layers.51.attn.indexer.compressor.wgate",
+      "layers.51.attn.indexer.compressor.wkv",
+      "layers.51.attn.indexer.compressor.gate_proj",
+      "layers.51.attn.indexer.compressor.kv_proj",
+      "layers.52.attn.compressor.wgate",
+      "layers.52.attn.compressor.wkv",
+      "layers.52.attn.compressor.fused_wkv_wgate",
+      "layers.52.attn.compressor.gate_proj",
+      "layers.52.attn.compressor.kv_proj",
+      "layers.52.attn.indexer.weights_proj",
+      "layers.52.attn.indexer.wq_b",
+      "layers.52.attn.indexer.q_b_proj",
+      "layers.52.attn.indexer.compressor.wgate",
+      "layers.52.attn.indexer.compressor.wkv",
+      "layers.52.attn.indexer.compressor.gate_proj",
+      "layers.52.attn.indexer.compressor.kv_proj",
+      "layers.53.attn.compressor.wgate",
+      "layers.53.attn.compressor.wkv",
+      "layers.53.attn.compressor.fused_wkv_wgate",
+      "layers.53.attn.compressor.gate_proj",
+      "layers.53.attn.compressor.kv_proj",
+      "layers.53.attn.indexer.weights_proj",
+      "layers.53.attn.indexer.wq_b",
+      "layers.53.attn.indexer.q_b_proj",
+      "layers.53.attn.indexer.compressor.wgate",
+      "layers.53.attn.indexer.compressor.wkv",
+      "layers.53.attn.indexer.compressor.gate_proj",
+      "layers.53.attn.indexer.compressor.kv_proj",
+      "layers.54.attn.compressor.wgate",
+      "layers.54.attn.compressor.wkv",
+      "layers.54.attn.compressor.fused_wkv_wgate",
+      "layers.54.attn.compressor.gate_proj",
+      "layers.54.attn.compressor.kv_proj",
+      "layers.54.attn.indexer.weights_proj",
+      "layers.54.attn.indexer.wq_b",
+      "layers.54.attn.indexer.q_b_proj",
+      "layers.54.attn.indexer.compressor.wgate",
+      "layers.54.attn.indexer.compressor.wkv",
+      "layers.54.attn.indexer.compressor.gate_proj",
+      "layers.54.attn.indexer.compressor.kv_proj",
+      "layers.55.attn.compressor.wgate",
+      "layers.55.attn.compressor.wkv",
+      "layers.55.attn.compressor.fused_wkv_wgate",
+      "layers.55.attn.compressor.gate_proj",
+      "layers.55.attn.compressor.kv_proj",
+      "layers.55.attn.indexer.weights_proj",
+      "layers.55.attn.indexer.wq_b",
+      "layers.55.attn.indexer.q_b_proj",
+      "layers.55.attn.indexer.compressor.wgate",
+      "layers.55.attn.indexer.compressor.wkv",
+      "layers.55.attn.indexer.compressor.gate_proj",
+      "layers.55.attn.indexer.compressor.kv_proj",
+      "layers.56.attn.compressor.wgate",
+      "layers.56.attn.compressor.wkv",
+      "layers.56.attn.compressor.fused_wkv_wgate",
+      "layers.56.attn.compressor.gate_proj",
+      "layers.56.attn.compressor.kv_proj",
+      "layers.56.attn.indexer.weights_proj",
+      "layers.56.attn.indexer.wq_b",
+      "layers.56.attn.indexer.q_b_proj",
+      "layers.56.attn.indexer.compressor.wgate",
+      "layers.56.attn.indexer.compressor.wkv",
+      "layers.56.attn.indexer.compressor.gate_proj",
+      "layers.56.attn.indexer.compressor.kv_proj",
+      "layers.57.attn.compressor.wgate",
+      "layers.57.attn.compressor.wkv",
+      "layers.57.attn.compressor.fused_wkv_wgate",
+      "layers.57.attn.compressor.gate_proj",
+      "layers.57.attn.compressor.kv_proj",
+      "layers.57.attn.indexer.weights_proj",
+      "layers.57.attn.indexer.wq_b",
+      "layers.57.attn.indexer.q_b_proj",
+      "layers.57.attn.indexer.compressor.wgate",
+      "layers.57.attn.indexer.compressor.wkv",
+      "layers.57.attn.indexer.compressor.gate_proj",
+      "layers.57.attn.indexer.compressor.kv_proj",
+      "layers.58.attn.compressor.wgate",
+      "layers.58.attn.compressor.wkv",
+      "layers.58.attn.compressor.fused_wkv_wgate",
+      "layers.58.attn.compressor.gate_proj",
+      "layers.58.attn.compressor.kv_proj",
+      "layers.58.attn.indexer.weights_proj",
+      "layers.58.attn.indexer.wq_b",
+      "layers.58.attn.indexer.q_b_proj",
+      "layers.58.attn.indexer.compressor.wgate",
+      "layers.58.attn.indexer.compressor.wkv",
+      "layers.58.attn.indexer.compressor.gate_proj",
+      "layers.58.attn.indexer.compressor.kv_proj",
+      "layers.59.attn.compressor.wgate",
+      "layers.59.attn.compressor.wkv",
+      "layers.59.attn.compressor.fused_wkv_wgate",
+      "layers.59.attn.compressor.gate_proj",
+      "layers.59.attn.compressor.kv_proj",
+      "layers.59.attn.indexer.weights_proj",
+      "layers.59.attn.indexer.wq_b",
+      "layers.59.attn.indexer.q_b_proj",
+      "layers.59.attn.indexer.compressor.wgate",
+      "layers.59.attn.indexer.compressor.wkv",
+      "layers.59.attn.indexer.compressor.gate_proj",
+      "layers.59.attn.indexer.compressor.kv_proj",
+      "layers.60.attn.compressor.wgate",
+      "layers.60.attn.compressor.wkv",
+      "layers.60.attn.compressor.fused_wkv_wgate",
+      "layers.60.attn.compressor.gate_proj",
+      "layers.60.attn.compressor.kv_proj",
+      "layers.60.attn.indexer.weights_proj",
+      "layers.60.attn.indexer.wq_b",
+      "layers.60.attn.indexer.q_b_proj",
+      "layers.60.attn.indexer.compressor.wgate",
+      "layers.60.attn.indexer.compressor.wkv",
+      "layers.60.attn.indexer.compressor.gate_proj",
+      "layers.60.attn.indexer.compressor.kv_proj"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {},
+    "transform_config": {},
+    "version": "0.15.1.a20260515",
+    "scale_fmt": "ue8m0"
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "compress": {
+      "attention_factor": 1.0,
+      "beta_fast": 32,
+      "beta_slow": 1,
+      "factor": 16,
+      "original_max_position_embeddings": 65536,
+      "partial_rotary_factor": 0.125,
+      "rope_theta": 160000,
+      "rope_type": "yarn",
+      "type": "yarn"
+    },
+    "main": {
+      "partial_rotary_factor": 0.125,
+      "rope_theta": 10000,
+      "rope_type": "default"
+    },
+    "partial_rotary_factor": 0.125,
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.5,
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "scoring_func": "sqrtsoftplus",
+  "sliding_window": 128,
+  "swiglu_limit": 10.0,
+  "tie_word_embeddings": false,
+  "topk_method": "noaux_tc",
+  "transformers_version": "5.8.1",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "compress_ratios": [
+    0,
+    0,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    0
+  ],
+  "num_hash_layers": 3,
+  "rope_scaling": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 16,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  },
+  "torch_dtype": "bfloat16"
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "transformers_version": "5.8.1"
+}

model-00001-of-00004.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:60e60fb974e48f8a1e20745bbbbd4336650e980ad0fba6a561e683a1e83424b0
+size 51194922826

model-00002-of-00004.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:26a130e128f6aaefd1673dd0ead54f685a0ca70597f81e550c2c8b21068a88c9
+size 50169748020

model-00003-of-00004.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:35eb79583b4577d630a269561ad53c010408dd8152a05c6e49b339047dbe909b
+size 50164611144

model-00004-of-00004.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4ee5ea80eac352f436fd93a358a02947724aa5785d5315fa73751241173bb0ed
+size 18814174990

model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

recipe.yaml ADDED Viewed

	@@ -0,0 +1,61 @@

+default_stage:
+  default_modifiers:
+    GPTQModifier:
+      config_groups:
+        attention:
+          targets: ['re:^model\.layers\.\d+\.self_attn\.(q_a_proj|q_b_proj|kv_proj|o_a_proj|o_b_proj)$',
+            're:^model\.layers\.\d+\.self_attn\.compressor\.(gate_proj|kv_proj)$', 're:^model\.layers\.\d+\.self_attn\.compressor\.indexer\.(gate_proj|kv_proj|q_b_proj|weights_proj)$']
+          weights:
+            num_bits: 8
+            type: float
+            symmetric: true
+            group_size: null
+            strategy: block
+            block_structure: [128, 128]
+            dynamic: false
+            actorder: null
+            scale_dtype: null
+            zp_dtype: null
+            observer: memoryless_minmax
+            observer_kwargs: {}
+          input_activations:
+            num_bits: 8
+            type: float
+            symmetric: true
+            group_size: 128
+            strategy: group
+            block_structure: null
+            dynamic: true
+            actorder: null
+            scale_dtype: null
+            zp_dtype: null
+            observer: null
+            observer_kwargs: {}
+          output_activations: null
+          format: null
+        experts:
+          targets: ['re:^model\.layers\.\d+\.mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$']
+          weights:
+            num_bits: 4
+            type: int
+            symmetric: true
+            group_size: 128
+            strategy: group
+            block_structure: null
+            dynamic: false
+            actorder: !!python/object/apply:compressed_tensors.quantization.quant_args.ActivationOrdering [
+              static]
+            scale_dtype: null
+            zp_dtype: null
+            observer: memoryless_minmax
+            observer_kwargs: {}
+          input_activations: null
+          output_activations: null
+          format: null
+      targets: [Linear]
+      ignore: [lm_head, 're:.*mtp\..*']
+      bypass_divisibility_checks: false
+      block_size: 128
+      dampening_frac: 0.1
+      actorder: static
+      offload_hessians: true

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}