Supermix v57 — MiMoMix Talk

A 3.08M-parameter decoder-only language model trained from scratch on CPU. It is the first trained checkpoint of the Supermix "MiMoMix" architecture, which until now existed only as untrained code.

Read this before using it. This model has a 582-token vocabulary and was trained on 21MB of templated coding-assistant dialogue. It produces fluent short replies in that one register and knows nothing else. It has no world knowledge, cannot follow instructions it has not seen phrased, and cannot produce a word outside its vocabulary. It is a research artifact demonstrating that this architecture trains and generates — not an assistant.

What it actually says

you:   hello
model: Hello. Tell me what you need and I will do my best to help.

you:   can you help me with tests
model: Understood. Yes. I can add unit tests and integration tests based on your code.

you:   why is my script failing
model: Check the traceback first, then we can isolate the failing function.

you:   what is your name
model: I am your local coding assistant for this project.

you:   write a unit test for login
model: Okay. I can do that. Share expected behavior and edge cases for login.

Every token is generated by the model. Nothing is templated at inference time.

Architecture

The model is MiMoMixModel from mimomix_core.py, a decoder-only transformer combining:

Hybrid attention — sliding-window layers interleaved with global layers (LLLG at 4 layers, 64-token window), each head carrying a learnable attention sink so it can place mass in a null slot instead of being forced to normalise over real tokens.
Auxiliary-loss-free sparse MoE — 8 routed experts, top-2, plus a shared expert. The router bias is updated by the sign rule once per optimizer step (not per forward, which would make the effective step size depend on gradient-accumulation depth and oscillate).
Multi-token prediction — 2 depths, reused at inference as a self-draft for speculative decoding.
A recursive thinking core — weight-tied latent refinement with PonderNet/ACT halting and a temperature-calibrated quality verifier.
Decoupled RoPE tables for local and global layers.

property	value
parameters	3,076,521 total / 1,292,829 active per token
layers	4 (`LLLG`), hidden 192, 6 heads, 2 KV heads
experts	8 routed, top-2, plus 1 shared
context	128 tokens
vocabulary	582 word types

Measured results

Held-out validation, 2,400 conversation pairs never trained on:

measurement	value
validation loss	0.2351 (0.339 bits/token)
perplexity	1.27 (uniform baseline 6.37)
held-out vocabulary coverage	1.0000
MTP acceptance length	2.429 — 7 trunk forwards vs 17 for plain greedy
speculative output identical to greedy	true
MoE routing entropy	1.000 normalised, 0 starved experts
serving throughput	55–75 tokens/second, CPU

Training: 3,000 steps, batch 16, sequence length 128 — about 6.1M tokens, ~1.2 passes over the corpus, 4,283 seconds on CPU. Full receipt in talk_results.json.

How to read the perplexity. 1.27 is a statement about the corpus at least as much as about the model. The training data is templated: only 37,543 of its 120,000 responses are distinct, so a validation row's response text can still appear in training. Rows are split disjointly, but the metric measures fit to a template distribution, not generalisation to unseen language. Do not compare it to perplexities reported on natural-text corpora.

Training data

databases/llm_chat.db from the Supermix repository — 120,000 (user, response) pairs of synthetic coding-assistant dialogue, 21.0M characters, 4.62M word tokens, and 292 distinct word types. Prompt tokens are masked out of the loss so the model learns to produce replies rather than echo the user.

This corpus is the model's ceiling. There is no world knowledge in it to learn, so there is none in the model.

Usage

The architecture is custom, not a transformers model, so the modules ship with the weights:

pip install torch huggingface_hub
python example.py

import torch, mimomix_decoding as decoding, mimomix_text as text_utils
from mimomix_core import MiMoMixConfig, MiMoMixModel

payload = torch.load("v57_talk_v2.pt", map_location="cpu", weights_only=False)
model = MiMoMixModel(MiMoMixConfig(**payload["config"]))
model.load_state_dict(payload["state_dict"])
model.eval()
tokenizer = text_utils.WordTokenizer.from_dict(payload["tokenizer"])

ids, _ = tokenizer.encode_turn("can you help me with tests", None)
out = decoding.speculative_generate(
    model, torch.tensor([ids]), max_new_tokens=48, eos_token_id=text_utils.EOS
)
print(tokenizer.decode(out.new_tokens[0].tolist()).strip())

The tokenizer is bundled inside the checkpoint. Weights loaded against a different vocabulary decode to confident nonsense rather than failing, so the two travel together.

Decoding

speculative_generate uses the MTP depths as a draft model. For greedy decoding this is provably token-identical to one-at-a-time generation, so it only changes cost — swap in decoding.greedy_generate and compare. For varied output, sample from the logits yourself; the bundled decoder is greedy by design, because that is what makes the equivalence guarantee checkable.

Limitations

582-token vocabulary. A word outside it cannot be generated, and your input words outside it become <unk>. Check with tokenizer.unknown_rate(text).
One register. Short coding-assistant replies. Ask about anything else and you will get a coding-assistant reply anyway.
No factual reliability. Nothing it says is grounded in anything. Fluent phrasing is not evidence of content.
128-token context.
No safety evaluation of generated text has been performed. The corpus is narrow and synthetic, which limits the surface, but no red-teaming was done.
Not comparable to modern language models. 3M parameters, under two hours on a CPU, 21MB of templated text.

What this does not demonstrate

That the architecture is validated as a language model. v57 shows the stack trains and generates on one small corpus. Scaling behaviour, long-context quality, and the value of the hybrid attention, the sparse MoE, or the thinking core at any real scale are all unmeasured here. No ablation has been run against a model without the thinking core on this corpus.

Citation and source

Part of the Supermix project. The architecture derives from the v53 MiMoMix line, which was in turn motivated by Xiaomi's published MiMo descriptions of hybrid attention, auxiliary-loss-free MoE balancing, and multi-token prediction. Those publications motivate the design and do not validate this implementation or its quality.

Design notes: docs/V57_TALKING_MIMOMIX.md and docs/V53_MIMOMIX_ARCHITECTURE.md in the source repository.

MIT licensed.

Downloads last month: 127