anemll/Qwen3.8-2.4T-A95B-FlashMoE-UD-Q1_0

Qwen3.8-2.4T-A95B Flash-MoE UD-Q1_0

SSD-backed Flash-MoE package for Qwen3.8-2.4T-A95B, exported from the Unsloth UD-Q1_0 GGUF.

This is not a conventional standalone GGUF. The non-routed and shared tensors are stored in model-dense.gguf; routed expert tensors are stored separately in a layer-major sidecar/ and loaded into a per-layer resident slot bank on demand. Use the Flash-MoE-enabled Qwen3.8 branch of anemll-flash-llama.cpp. Stock llama.cpp builds that do not understand this package layout or its IQ1_XXXS routed experts cannot run it.

Model and package details

Property	Value
Architecture	qwen35moe
Transformer blocks	93
Embedding width	8192
Context metadata	262,144 tokens
Routed experts per layer	512
Native experts selected per token	10
Dense/shared GGUF tensors	1,423
Dense/shared GGUF file size	36,881,771,680 bytes
Routed tensors	279
Routed sidecar size	360,374,599,680 bytes
Complete upload size	397,256,650,449 bytes (369.97 GiB)

The routed sidecar preserves the source GGUF bytes exactly:

• Layers 0-91: gate, up, and down routed tensors are IQ1_XXXS (276 tensors).
• Layer 92: gate, up, and down routed tensors are Q2_K (3 tensors).
• Each layer file contains the three whole routed tensors for that layer, with 512 experts packed within each tensor.

The dense/shared GGUF quant inventory is:

Quant type	Tensors	Tensor-data bytes
F32	838	2,171,133,440
Q4_K	2	2,288,517,120
Q5_K	420	23,391,633,408
Q6_K	162	8,876,851,200
Q8_0	1	142,606,336

Download

The repository is about 397 GB. Put it on a fast local SSD and preserve the directory layout:

hf download anemll/Qwen3.8-2.4T-A95B-FlashMoE-UD-Q1_0 \
  --local-dir ~/Models/Qwen3.8

Expected layout:

~/Models/Qwen3.8/
├── model-dense.gguf
├── flashmoe-package.json
└── sidecar/
    ├── manifest.json
    ├── layer_000.bin
    ├── ...
    └── layer_092.bin

Keep the sidecar on fast directly attached storage. This runtime uses SSD reads for cache misses; network filesystems and slow external links will sharply reduce decode speed.

SSD slot-bank inference

Build the Flash-MoE branch of anemll-flash-llama.cpp, then run from its source directory:

./build/bin/llama-cli \
  -m ~/Models/Qwen3.8/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/Qwen3.8/sidecar \
  --moe-slot-bank 96 \
  --moe-topk 10 \
  --moe-cache-io-split 4 \
  --moe-prefetch-temporal \
  -fit on \
  -ub 1 \
  -b 1 \
  -ngl 999 \
  -c 128 \
  --no-warmup \
  -st \
  -p "Hello" \
  -n 16

Important runtime choices:

• Keep --moe-topk 10: this is the model's native routed width.
• Do not add --slot4 or --slot8; those fused paths do not match native top-10 routing.
• -ub 1 and -b 1 select the decode-oriented SSD path.
• --moe-prefetch-temporal overlaps next-layer reads with current-layer compute.
• The 96-slot setting is intended for a 128 GB Apple Silicon machine. Each slot, summed over all 93 layer banks, costs 703,856,640 bytes (671.25 MiB), so a 96-slot bank uses about 62.93 GiB in addition to the dense model and runtime allocations. If memory pressure is high, start with 32 or 64 slots.
• The 262K value is model metadata, not a promise that the full context fits in memory. Increase -c only after budgeting KV-cache and runtime memory.

Attribution and license

• Base model: Qwen/Qwen3.8-2.4T-A95B
• Source quantization: unsloth/Qwen3.8-2.4T-A95B-GGUF, UD-Q1_0
• Flash-MoE packaging/runtime: ANEMLL anemll-flash-llama.cpp, Qwen3.8 branch

The embedded GGUF metadata reports license: other. Review and comply with the base model and source repository license terms before use or redistribution.

Verification metadata

flashmoe-package.json records the export summary and sidecar/manifest.json records source offsets, quant types, shapes, exact byte lengths, and repacked offsets. The package was produced with byte-level sidecar verification enabled.