---
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
  - custom_code
  - tiny
  - looped
  - text-generation
---

# min-spark

min-spark is a ~5.76M-parameter decoder-only language model, trained on
10.01B tokens (WSD-decayed). It is the first release in the PICO series: a
looped, weight-shared hybrid with a tied byte-level BPE vocabulary of 4,096.

Three inference settings — **effort levels** — trade compute for quality:

| Effort | Compute |
|--------|---------|
| min-spark-low | quickest |
| min-spark-medium | balanced (default) |
| min-spark-high | highest quality |

## Quick start

> **Private repo:** the repository is currently private. Authenticate first:
> `hf auth login` (or set `HF_TOKEN`). This becomes unnecessary at public release.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark", trust_remote_code=True
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark", trust_remote_code=True
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, effort="high", max_new_tokens=64)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

With `pipeline`:

```python
from transformers import pipeline
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe(prompt, effort="high", max_new_tokens=64)[0]["generated_text"])
```

> **Generation is single-sequence.** Batched generation and left padding are
> not supported (right-padded batching works for scoring, e.g. lm-eval).
> **No KV cache** in this release — it arrives in min-spark 1.1.

Without Transformers, use the bundled `generate.py`:
```bash
python generate.py -p "The meaning of life is" -e high
```

## Benchmarks

Chart and table generated from `results_all_k.json` (exact values):

![accuracy](charts/accuracy.svg)
![byte perplexity](charts/byteppl.svg)

<details>
<summary>Benchmark table (source of truth)</summary>

| Effort | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | WikiText-2 byte-ppl |
|--------|-------|----------|---------------|-----------|------|---------------------|
| min-spark-low | 67.11% | 35.10% | 23.21% | 27.91% | 54.13% | 2.8783 |
| min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
| min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |

</details>

> These are the exact values from `results_all_k.json`. If they drift from
> `charts/benchmark_table.md` (the source of truth), the table here is wrong.

Metrics: BLiMP acc, ARC-Easy/ARC-Challenge/HellaSwag/PIQA acc_norm,
WikiText-2 byte perplexity (lower is better), evaluated with the
lm-eval==0.4.12 harness (batched, masked/train-consistent; the published
methodology — see `run_lmeval.py`).

### lm-eval notes

- Reproduce the published numbers: `python run_lmeval.py --effort medium`
  (add `--limit N` to bound runtime).
- The stock `--model hf` path also works and is measured to differ from the
  published methodology as recorded in `deltas.json` (paired same-subset
  measurement). The published numbers are the methodology in `run_lmeval.py`.

## Architecture

The native Hugging Face architecture viewer was investigated (see
`ARCHITECTURE-VIEWER.md`); the robust fallback diagram below ships regardless.

![architecture](architecture.svg)

## Technical note

min-spark is Meiosis (PICO release 1), a tied-embedding looped decoder:
embed → prelude ×1 → body of 3 distinct GQA blocks (6 Q / 2 KV heads, FFN 768)
× K loops, with per-loop LoRA (rank 16), a transient loop embedding, and
Deep-Delta (vdim 1) residuals → coda ×1 → RMSNorm → tied unembed. 11 block
applications per token. Effort maps to the internal loop count K: low=2,
medium=3, high=4. Trained 10.01B tokens on a filtered fineweb-edu +
finemath-4plus mix with a warmup-stable-decay schedule and a 20% cooldown.

## Provenance

- Source checkpoint: `meiosis-decay-p09`, step 182,875, 10.01B tokens.
- `model.safetensors` sha256: `a121c6a316e54e490918d93d3d265460d45addbdad49b8448c514e5f61232c41`
- `tokenizer.json` sha256: `d671f013ebdb06a55e48d047f12621c3311310fdcc344683d434d294a909ac51`
- Eval: lm-eval==0.4.12, batched (batch 32), masked/train-consistent; the
  per-effort loop-count mapping and best-of selection are in the Technical
  note.

## License

Apache-2.0. See [LICENSE](LICENSE).