license: other
license_name: minimax-music3-terms
license_link: https://huggingface.co/MiniMaxAI/MiniMax-Music3
tags:
- minimax-music3
- composer-lm
- depth-pruning
- distillation
MiniMax Music3 composer LM — 5.7B depth-pruned distilled students (LR ablation pair)
The Music3 composer (Qwen3-8B post-trained by MiniMax to emit RVQ audio codes) depth-pruned from 36 to 21 layers (5.69B params) and repair-distilled against the frozen teacher. Both checkpoints here are the same recipe differing ONLY in learning rate — an A/B pair, both of which pass listening tests (recognizable genre, structure, and vocals on held-out prompts) while composing ~2x faster than the teacher (single-row / CFG-free: the distillation targets are the CFG-guided distributions, so guidance is baked in and no unconditional pass is needed).
Listen — same prompt, same seed, same renderer (UK grime, 40 s)
Only the composer differs. Teacher = stock 8B dual-row CFG path; students compose single-row, ~2x faster.
Teacher (8B, reference):
Student 5.7B — lr 3e-5 arm:
Student 5.7B — lr 6e-5 arm:
| subfolder | LR | notes |
|---|---|---|
lr-3e-5/ |
3e-5 | conservative anneal arm |
lr-6e-5/ |
6e-5 | doubled-LR arm |
Recipe: ShortGPT-style layer scoring over a 18.7k-track self-distillation corpus (Mothersuperior/minimax-music3-rvq-distill-corpus-8k), drop 15/36 layers (first/last two protected, consecutive-run cap 3), then repair with CE vs teacher top-50 guided distributions + teacher-normalized hidden-state MSE (final-layer, cached fp16 targets) + END-token supervision. bf16, torch-optimi AdamW, batch 8, single RTX PRO 6000. Hidden-state cosine to teacher ~0.94 at upload (teacher self-consistency ceiling 0.99994).
Drop-in: standard Qwen3ForCausalLM.from_pretrained(...), same embeddings/width — compatible with
the stock Music3 depth decoder + DiT. Compose single-row (sample top-50 from masked logits; no CFG).
Use is subject to the MiniMax Music 3 model terms.