open-rvq-encoder-minimax-music3-169m-53k
Audio -> RVQ-codes encoder for MiniMax-Music3, replacing the unreleased official tokenizer. SimpleTuner's v4 architecture (169M: 1088-wide shared encoder + causal depth decoder across the 8 RVQ books), reimplemented parameter-exact and trained from scratch on a 53k-track (900 h) self-distilled corpus — 60 s tracks minted through the official diffusers pipeline, with a deliberate focus on non-English vocals (German, French, Italian, Russian, Polish, and more). Both public community corpora are English-only; this is currently the only encoder trained on multilingual vocal material.
Results (own holdout — random-crop, ~1.6k tracks)
| metric | value |
|---|---|
| semantic top-1 | 54.51% |
| acoustic top-1 (teacher-forced) | 29.91% |
| acoustic top-1 (free-running) | 11.19% |
All numbers are from this project's own holdout and evaluation code
(history.json included); holdout construction differs across community
projects, so no cross-model comparison is claimed here — run your own
evals and draw your own conclusions.
Other findings measured within this project, same caveat: at matched 53k data, a 31M hiddens-regression encoder reached parity with this architecture until the LR anneal, after which causal-depth led by ~2-3 sem points; this model scores 52.2% sem zero-shot on the Mothersuperior diversity corpus, and continued training pooled with it moved semantic accuracy by ~0.
Usage
Self-contained loader, no external training framework required:
import torch
from train_v4 import V4Encoder, frame_latent_starts, pool_matrix
model = V4Encoder().eval()
model.load_state_dict(
torch.load("v4_best.pt", map_location="cpu", weights_only=False)["model"])
# inputs: DAV latents z [128, L] from the official dav.pth encoder,
# windowed to 128 frames via the stitched-timeline alignment
# (200-frame DiT windows / 100 hop / 345-latent hop / ownership from
# frame 25) -- frame_latent_starts() implements it exactly.
# feats, sem_logits = model(latents[B,448,128], pool[B,128,448])
# c0 = sem_logits.argmax(-1); c1..c7 = model.depth.greedy(feats, c0)
Emitted-frame convention: code row 0 is the un-emitted priming row;
frame i corresponds to codes[i+1].
Training deviations from SimpleTuner v4 (architecture untouched)
Own 53k corpus; no teacher-KL term; plain Linear readouts instead of MuReadout (identical at fixed width); cosine LR anneal instead of polynomial; validation on both center and random crops; stray-EOS semantic rows masked (ignore_index), not clamped.
Credit
Architecture: bghira / SimpleTuner. Alignment rules and evaluation conventions follow the community thread at MiniMaxAI/MiniMax-Music3 (bghira, Serveurperso, Mothersuperior, scragnog). All training audio is MiniMax-Music3 output; use is subject to the MiniMax Music 3 model terms.