Zenyx V3 Base (1.5B Mixture-of-Experts)
Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.
This is a BASE model — it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue (
"The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.
Current checkpoint: step 63,200 · 34.4B tokens seen
Model Architecture
- Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 active per token, with a Sinkhorn transport-based gate.
- Multi-head Latent Attention (MLA): compresses the KV cache into a low-rank latent subspace, cutting HBM bandwidth and memory footprint.
- Hyper-Connections: Sinkhorn-normalised residual routing for gradient stability at scale.
- Multi-Token Prediction (MTP): one auxiliary prediction head during training.
- Context: pretrained at up to 4,096 tokens (progressive 2,048 → 4,096). YaRN and RoPE scaling factors are precomputed so context can be extended at inference time beyond the trained length.
| Total parameters | ~1.5B |
| Active parameters / token | ~0.4B |
| Layers | 16 (2 dense + 14 MoE) |
| Hidden size | 1,536 |
| Attention heads | 12 (head dim 128) |
| Vocabulary | 129,280 |
| Precision | bfloat16 |
Benchmarks — checkpoint step 63,200 (34.4B tokens)
All tasks are evaluated with the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is taken as the answer. Nothing is generated and no output parsing is involved, so the numbers do not depend on instruction-following ability. 0-shot, full evaluation sets, no subsampling.
acc_norm normalises each continuation's log-likelihood by its length in characters,
which removes the bias toward short answers; it is the headline metric wherever the
task has candidates of differing lengths.
| Benchmark | acc |
acc_norm |
Random | Δ | n | Description |
|---|---|---|---|---|---|---|
| HellaSwag | 29.58% ± 0.46 | 32.22% ± 0.47 | 25.0% | +7.2 | 10,042 | Commonsense sentence completion |
| ARC-Easy | 49.54% ± 1.03 | 44.53% ± 1.02 | 25.0% | +19.5 | 2,376 | Grade-school science questions |
| ARC-Challenge | 20.90% ± 1.19 | 25.77% ± 1.28 | 25.0% | +0.8 | 1,172 | Hard grade-school science questions |
| PIQA | 60.83% ± 1.14 | 59.79% ± 1.14 | 50.0% | +9.8 | 1,838 | Physical commonsense reasoning |
| WinoGrande | 49.49% ± 1.40 | — | 50.0% | -0.5 | 1,267 | Pronoun resolution / coreference |
| OpenBookQA | 17.40% ± 1.70 | 30.00% ± 2.05 | 25.0% | +5.0 | 500 | Elementary science with open book |
| BoolQ | 60.55% ± 0.85 | 62.11% ± 0.85 | 62.2% | -1.6 | 3,270 | Yes/no reading comprehension |
| SciQ | 75.10% ± 1.37 | 68.50% ± 1.47 | 25.0% | +50.1 | 1,000 | Science exam questions with support |
| LAMBADA (OpenAI) | 25.50% ± 0.61 | — | 0.0% | +25.5 | 5,153 | Long-range last-word prediction |
Bold marks the metric that is conventional for that task — acc_norm for
HellaSwag, ARC, PIQA and OpenBookQA; acc for WinoGrande, BoolQ, SciQ and
LAMBADA. The convention is applied per task, not chosen per result: it lowers
the reported figure for ARC-Easy (44.53 rather than 49.54) and PIQA (59.79
rather than 60.83). Δ compares the bolded metric to the random baseline.
Both metrics
| Benchmark | acc |
acc_norm |
n |
|---|---|---|---|
| HellaSwag | 29.58% ± 0.46 | 32.22% ± 0.47 | 10,042 |
| ARC-Easy | 49.54% ± 1.03 | 44.53% ± 1.02 | 2,376 |
| ARC-Challenge | 20.90% ± 1.19 | 25.77% ± 1.28 | 1,172 |
| PIQA | 60.83% ± 1.14 | 59.79% ± 1.14 | 1,838 |
| WinoGrande | 49.49% ± 1.40 | — | 1,267 |
| OpenBookQA | 17.40% ± 1.70 | 30.00% ± 2.05 | 500 |
| BoolQ | 60.55% ± 0.85 | 62.11% ± 0.85 | 3,270 |
| SciQ | 75.10% ± 1.37 | 68.50% ± 1.47 | 1,000 |
| LAMBADA (OpenAI) | 25.50% ± 0.61 | — | 5,153 |
Language modelling
| Corpus | Value | Metric |
|---|---|---|
| WikiText-2 (raw) | 26.47 | token-level perplexity |
| WikiText-2 (raw) | 49.70 | word-level perplexity |
| WikiText-2 (raw) | 1.0509 | bits per byte |
| LAMBADA | 35.79 | perplexity of the target word |
WikiText-2 is scored with a rolling 1024-token window at stride 512, so every counted token is predicted with at least 512 tokens of left context and each token is counted exactly once. (Scoring disjoint windows instead inflates these figures by ~15% because the leading tokens of each window are predicted from nothing.)
Method validation
SciQ places the correct answer at a fixed index (option 4), following lm-evaluation-harness. Log-likelihood scoring is position-blind in principle, so the suite was re-run with the option order shuffled per document as a control:
| SciQ variant | acc |
acc_norm |
|---|---|---|
| answer at fixed index | 75.10% | 68.50% |
| option order shuffled | 76.00% | 68.30% |
The two differ by 0.9 points against a standard error of 1.37, i.e. 0.7σ, confirming the score reflects answer content rather than position. The same comparison bounds the effect of MoE expert-capacity variation across batches at under one point, since both runs score an identical set of (context, continuation) pairs and differ only in batching order.
Reading these numbers. This is a partially-trained 1.5B base model, so knowledge-heavy multiple-choice tasks sit close to their random baselines — that is expected at this scale and token count. The signal to watch is the language-modelling side: LAMBADA accuracy and WikiText perplexity measure whether the model has actually learned to predict text, and those improve steadily long before multiple-choice benchmarks move. Note also that BoolQ's majority-class baseline is 62.2%, so a score near that is not evidence of comprehension.
Hardware Serving Benchmarks (NVIDIA L4, 24 GB)
Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.
| Metric | Value | Notes |
|---|---|---|
| Decode speed | 68.5 tok/s | steady-state autoregressive decode |
| Warm prefill | ~20 ms | short prompt, shape already compiled |
| Checkpoint load | ~26 s | params → GPU, from local cache |
| Active VRAM | ~5.0 GB | of 24 GB |
Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.
Inference Example
from zenyx_v3_inference import ZenyxGenerator
generator = ZenyxGenerator(step=63200)
# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(generator.generate(
"The capital of France is",
max_new_tokens=80,
temperature=0.7,
repetition_penalty=1.15,
))
Evaluation Reproducibility
Benchmarks were produced by modal_base_evals.py on a single NVIDIA L4, scoring
continuations in batches with length-bucketed padding. Task formats follow the
lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions
and answer-key handling), so the numbers are broadly comparable to published
base-model results, though this is an independent implementation rather than a
harness run.
Limitations
- Pretraining is incomplete — the model will change substantially with more tokens.
- Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
- Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.