hayulalab/new-equation-adapter-moe-paper

The New Equation: Dense AdapterMoE — Dynamic Expert Routing on Consumer Hardware — Hayula Research

Hayula AI Lab

Abstract

Mixture-of-Experts models achieve efficient scaling by routing tokens to specialized sub-networks. However, traditional MoE requires training from scratch or expensive full-model fine-tuning. We propose Dense AdapterMoE: a hybrid architecture combining a frozen 7-8B base language model with 10-20 dynamically routed LoRA expert adapters, all loaded simultaneously on consumer hardware (Apple M2 Ultra, 192GB unified memory). Each expert adapter occupies only 37MB (2 million parameters at rank 8), a

Files

File	Description
paper.md	Full paper (Markdown)
README.md	This model card

Citation

@techreport{hayulalab2026newequationadaptermoe,
    title={The New Equation: Dense AdapterMoE — Dynamic Expert Routing on Consumer Hardware — Hayula Research},
    author={Hayula AI Lab},
    year={2026},
    url={https://huggingface.co/hayulalab/new-equation-adapter-moe-paper}
}

---

hayulalab — Open Source AI Research