---
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
  - custom_code
  - tiny
  - looped
  - text-generation
  - sub-10m
---

# min-spark

min-spark is a 5.76M-parameter decoder-only language model trained on 10.01B
tokens. It is the first release in the min-spark family of sub-10M models.

The model runs a small weight-shared core repeatedly instead of stacking many
distinct layers. Each pass through the core is one *loop*. The number of loops
is a dial you control at inference time: fewer loops is faster, more loops is
more capable. We expose three presets, called **effort levels**.

| Effort | Compute | Use when |
|--------|---------|----------|
| min-spark-low | quickest | high-throughput, short replies |
| min-spark-medium | balanced | general use (default) |
| min-spark-high | highest quality | you want the best answer per token |

## How to use

The model loads with standard Transformers. The repository is currently
private, so authenticate first with `hf auth login` (or set `HF_TOKEN`).

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark", trust_remote_code=True
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark", trust_remote_code=True
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, effort="high", max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

With `pipeline`:

```python
from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe(prompt, effort="high", max_new_tokens=64)[0]["generated_text"])
```

Two constraints to know up front.

- **Single sequence.** Generation runs one sequence at a time. Batched
  generation and left padding are not supported; right-padded batches work
  for scoring, which is what lm-eval uses.
- **No KV cache yet.** It arrives in min-spark 1.1. For a 5.76M model this is
  rarely a bottleneck, but long generations re-encode the prefix each step.

A transformers-free path is bundled as `generate.py`:

```bash
python generate.py -p "The meaning of life is" -e high
```

## Benchmarks

Scores are zero-shot, evaluated with lm-eval 0.4.12, batched and
masked/train-consistent. BLiMP is accuracy; ARC-Easy, ARC-Challenge,
HellaSwag, and PIQA are length-normalized accuracy; WikiText-2 is byte
perplexity, lower is better.

Across the three effort levels, min-spark sits in the mid-60s on BLiMP,
low-to-mid 30s on ARC-Easy, and low-20s on ARC-Challenge and HellaSwag. On
PIQA it lands in the low-50s. The best WikiText-2 byte perplexity is 2.77.
Higher effort helps on BLiMP, where medium and high reach about 69; on the
other tasks the three levels stay close, which means the common-sense tasks
are less sensitive to loop count than grammar is.

### By effort level

![min-spark accuracy by effort](charts/effort.svg)

| Effort | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | WikiText-2 byte-ppl |
|--------|-------|----------|---------------|-----------|------|---------------------|
| min-spark-low | 67.11% | 35.10% | 23.21% | 27.91% | 54.13% | 2.8783 |
| min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
| min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |

### Against other small models

The chart below takes the best score min-spark reaches on each benchmark
across effort levels, and compares it to published results from comparable
sub-10M models: [GPT-S2-5M](https://huggingface.co/AxiomicLabs/GPT-S2-5M),
[SLM-10M](https://huggingface.co/LiodonAI/SLM-10M), and
[michel-nano-v2](https://huggingface.co/finnianx/michel-nano-v2). Scores are
from those models' published cards, so some cells are blank where a model does
not report that benchmark.

![min-spark best score vs small-model peers](charts/comparison.svg)

min-spark matches or beats the comparison set on ARC-Easy (37.1 vs the next
best 35.9) and ARC-Challenge (23.2 vs 23.5 for SLM-10M, above the rest). On
HellaSwag and PIQA it sits in the same band as the peers. BLiMP is reported
only by michel-nano-v2 (72.5), where min-spark lands at 69.2.

### Against KeyLM

The prior release in this line, [KeyLM-75M](https://huggingface.co/MinimaLabs/KeyLM-75M),
is a 75.25M-parameter model trained on 18B tokens, thirteen times the size of
min-spark. min-spark reaches roughly comparable common-sense numbers at a
fraction of the parameter budget. KeyLM reports ARC at 29.9 average and
HellaSwag at 29.7; min-spark reaches 37.1 on ARC-Easy, 23.2 on ARC-Challenge,
and 27.9 on HellaSwag. On PIQA, KeyLM scores 60.0 and min-spark reaches 54.4.

The gap on PIQA and the ARC average is the honest picture: the looped
architecture compresses a lot of the parameter-count advantage, but a 13x
parameter difference still shows up on the harder relational tasks. What the
comparison does show is that a 5.76M looped model is in the same general band
as a 75M standard model on a large share of these tasks, at less than a tenth
of the inference cost. KeyLM's reported numbers are zero-shot accuracy from
its own card.

## Architecture

min-spark is a tied-embedding, looped decoder. A byte-level BPE vocabulary of
4,096 feeds an embedding, a single prelude block, then a body of three
distinct grouped-query-attention blocks that are run K times, then a coda
block and a tied unembedding. Each loop pass adds a per-loop LoRA and a
transient loop embedding; residuals use Deep-Delta projections. There are 11
block applications per token.

The model runs 11 block applications per token, which is why a 5.76M
parameter count behaves more like a somewhat larger standard model.

The interactive architecture graph is rendered by
[hfviewer](https://hfviewer.com/), a graph viewer that traces the model and
lets you zoom from the top-level shape down to individual blocks. The embed
below is a summary card; open it to reach the full graph with block-level
detail. The first render can take a few minutes while hfviewer traces the
model.

[![min-spark architecture](https://hfviewer.com/api/card.svg?source=MinimaLabs%2Fmin-spark&granularity=auto&animated=false)](https://hfviewer.com/MinimaLabs/min-spark)

The three effort levels map to the internal loop count K: min-spark-low is
K=2, min-spark-medium is K=3, min-spark-high is K=4.

## Training

min-spark was trained on 10.01B tokens drawn from a filtered fineweb-edu and
finemath-4plus mix, with a warmup-stable-decay schedule and a 20% cooldown.
The released checkpoint is at step 182,875 of that run.

`model.safetensors` sha256:
`a121c6a316e54e490918d93d3d265460d45addbdad49b8448c514e5f61232c41`

`tokenizer.json` sha256:
`d671f013ebdb06a55e48d047f12621c3311310fdcc344683d434d294a909ac51`

## Reproducing the benchmarks

The bundled `run_lmeval.py` reproduces the published numbers exactly. Run it
with the effort you want to check:

```bash
python run_lmeval.py --effort medium --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext
```

Add `--limit N` to bound runtime. The stock `--model hf` path in lm-eval also
works; the measured difference between it and the published methodology is
recorded in `deltas.json`.

## Limitations

- A 5.76M model is small. Factual recall, multi-step reasoning, and
  long-range coherence are weak.
- The base model is not instruction-tuned, aligned, or filtered for safety.
  It completes text as-is, including harmful or offensive continuations.
- Context length is 512 tokens.
- Generation is single-sequence, and there is no KV cache yet.
- Scores were collected at a 512-token context; longer contexts are not
  validated.

## License

Apache-2.0. See [LICENSE](LICENSE).