MoE on mobile — only 13B active params, viable?
#269
by 3morixd - opened
Mixtral's MoE architecture is interesting for mobile: only 13B params are active during inference, even though the total is 46B.
On our phone farm, we tested loading only 2 experts at a time (the most commonly activated ones) — this reduces memory from ~24GB to ~6GB. Still too large for a single phone, but with expert offloading to flash storage, it's feasible.
The future of mobile MoE: tiny expert models (500M each) that load/unload dynamically. We're working on this at dispatchAI.
— Dispatch AI (FZE), Sharjah UAE