---
license: apache-2.0
task_categories:
- image-classification
tags:
- mixture-of-experts
- sparse-routing
- load-balancing
- pytorch
---

# Pocket MoE

Pocket MoE is a compact top-2 mixture-of-experts classifier. A shared image encoder
feeds four specialist MLPs while a learned router activates two experts for each
example.

The benchmark compares an MoE and a parameter-matched dense MLP on the same split,
then reports expert utilization, routing entropy, and each digit's dominant expert.

## Reproduce

```powershell
uv run python projects/tiny-vision-foundry/prepare_data.py
uv run python projects/pocket-moe/train.py
```

Sparse routing here demonstrates conditional computation; it does not reduce the
stored checkpoint size because all four experts remain available.

## Verified results

| Model | Stored parameters | Active parameters | Test accuracy | Macro F1 |
| --- | ---: | ---: | ---: | ---: |
| Dense control | 5,490 | 5,490 | 96.30% | 0.9618 |
| Top-2 Pocket MoE | 5,004 | **3,608** | 96.30% | **0.9625** |

The MoE activates 34.3% fewer parameters per example than the dense control without
losing accuracy. Experts specialized strongly: digit `0` routed 86.5% to expert 3,
digit `2` routed 83.1% to expert 2, and digit `7` routed 90.9% to expert 0.

Utilization was not perfectly balanced: expert shares were 44.2%, 12.4%, 19.6%, and
23.8%, for a coefficient of variation of 0.473. The stored MoE still includes all four
experts, so conditional compute should not be confused with checkpoint compression.