---
license: other
base_model:
  - Qwen/Qwen3.8-2.4T-A95B
tags:
  - gguf
  - llama.cpp
  - qwen3.8
  - moe
  - flash-moe
  - ssd
  - apple-silicon
  - text-generation
pipeline_tag: text-generation
library_name: llama.cpp
---

# Qwen3.8-2.4T-A95B Flash-MoE UD-Q1_0

SSD-backed Flash-MoE package for **Qwen3.8-2.4T-A95B**, exported from the
[Unsloth UD-Q1_0 GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/tree/main/UD-Q1_0).

This is not a conventional standalone GGUF. The non-routed and shared tensors
are stored in `model-dense.gguf`; routed expert tensors are stored separately in
a layer-major `sidecar/` and loaded into a per-layer resident slot bank on demand.
Use the Flash-MoE-enabled [`Qwen3.8` branch of
`anemll-flash-llama.cpp`](https://github.com/Anemll/anemll-flash-llama.cpp/tree/Qwen3.8).
Stock `llama.cpp` builds that do not understand this package layout or its
`IQ1_XXXS` routed experts cannot run it.

## Model and package details

| Property | Value |
|---|---:|
| Architecture | `qwen35moe` |
| Transformer blocks | 93 |
| Embedding width | 8192 |
| Context metadata | 262,144 tokens |
| Routed experts per layer | 512 |
| Native experts selected per token | 10 |
| Dense/shared GGUF tensors | 1,423 |
| Dense/shared GGUF file size | 36,881,771,680 bytes |
| Routed tensors | 279 |
| Routed sidecar size | 360,374,599,680 bytes |
| Complete upload size | 397,256,650,449 bytes (369.97 GiB) |

The routed sidecar preserves the source GGUF bytes exactly:

- Layers 0-91: gate, up, and down routed tensors are `IQ1_XXXS` (276 tensors).
- Layer 92: gate, up, and down routed tensors are `Q2_K` (3 tensors).
- Each layer file contains the three whole routed tensors for that layer, with
  512 experts packed within each tensor.

The dense/shared GGUF quant inventory is:

| Quant type | Tensors | Tensor-data bytes |
|---|---:|---:|
| `F32` | 838 | 2,171,133,440 |
| `Q4_K` | 2 | 2,288,517,120 |
| `Q5_K` | 420 | 23,391,633,408 |
| `Q6_K` | 162 | 8,876,851,200 |
| `Q8_0` | 1 | 142,606,336 |

## Download

The repository is about 397 GB. Put it on a fast local SSD and preserve the
directory layout:

```bash
hf download anemll/Qwen3.8-2.4T-A95B-FlashMoE-UD-Q1_0 \
  --local-dir ~/Models/Qwen3.8
```

Expected layout:

```text
~/Models/Qwen3.8/
├── model-dense.gguf
├── flashmoe-package.json
└── sidecar/
    ├── manifest.json
    ├── layer_000.bin
    ├── ...
    └── layer_092.bin
```

Keep the sidecar on fast directly attached storage. This runtime uses SSD reads
for cache misses; network filesystems and slow external links will sharply reduce
decode speed.

## SSD slot-bank inference

Build the Flash-MoE branch of `anemll-flash-llama.cpp`, then run from its source
directory:

```bash
./build/bin/llama-cli \
  -m ~/Models/Qwen3.8/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/Qwen3.8/sidecar \
  --moe-slot-bank 96 \
  --moe-topk 10 \
  --moe-cache-io-split 4 \
  --moe-prefetch-temporal \
  -fit on \
  -ub 1 \
  -b 1 \
  -ngl 999 \
  -c 128 \
  --no-warmup \
  -st \
  -p "Hello" \
  -n 16
```

Important runtime choices:

- Keep `--moe-topk 10`: this is the model's native routed width.
- Do **not** add `--slot4` or `--slot8`; those fused paths do not match native
  top-10 routing.
- `-ub 1` and `-b 1` select the decode-oriented SSD path.
- `--moe-prefetch-temporal` overlaps next-layer reads with current-layer compute.
- The 96-slot setting is intended for a 128 GB Apple Silicon machine. Each slot,
  summed over all 93 layer banks, costs 703,856,640 bytes (671.25 MiB), so a
  96-slot bank uses about 62.93 GiB in addition to the dense model and runtime
  allocations. If memory pressure is high, start with 32 or 64 slots.
- The 262K value is model metadata, not a promise that the full context fits in
  memory. Increase `-c` only after budgeting KV-cache and runtime memory.

## Attribution and license

- Base model: [Qwen/Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B)
- Source quantization: [unsloth/Qwen3.8-2.4T-A95B-GGUF, UD-Q1_0](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/tree/main/UD-Q1_0)
- Flash-MoE packaging/runtime: [ANEMLL `anemll-flash-llama.cpp`, `Qwen3.8`
  branch](https://github.com/Anemll/anemll-flash-llama.cpp/tree/Qwen3.8)

The embedded GGUF metadata reports `license: other`. Review and comply with the
base model and source repository license terms before use or redistribution.

## Verification metadata

`flashmoe-package.json` records the export summary and `sidecar/manifest.json`
records source offsets, quant types, shapes, exact byte lengths, and repacked
offsets. The package was produced with byte-level sidecar verification enabled.