---
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
  - custom_code
  - tiny
  - looped
  - text-generation
  - sub-10m
---

![min-spark](charts/banner.svg)

# min-spark

**min-spark is a 5.76M-parameter language model with native effort levels.** It introduces controllable depth to the sub-10M model class. The same checkpoint can produce a quick completion or spend more computation on the same prompt, selected with one inference argument.

The model was trained on 10.01B tokens. Its looped decoder reuses a compact transformer core across multiple passes, giving a small model the computation of a deeper network while keeping the parameter count fixed.

## Introducing native effort levels

Effort levels have traditionally been associated with large reasoning models, where they adjust a thinking budget or modify the surrounding prompt. min-spark brings the idea directly into the language model. Each effort level changes the model's internal computation by selecting a different number of passes through its shared core.

| Effort | Character | Recommended use |
|---|---|---|
| `low` | Fastest | High-throughput completion |
| `medium` | Balanced | General generation |
| `high` | Most compute | Highest available quality |

Effort is a generation-time choice. All three levels use the same weights and tokenizer.

## Usage

min-spark is compatible with Transformers and requires remote code loading.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    effort="high",
    max_new_tokens=64,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

The same model works with the Transformers pipeline API:

```python
from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe(prompt, effort="high", max_new_tokens=64)
print(result[0]["generated_text"])
```

A lightweight Transformers-free generation script is included in the repository:

```bash
python generate.py -p "The meaning of life is" -e high
```

Generation currently runs one sequence at a time. Right-padded batches are supported for evaluation. KV caching is planned for a future release, so long generations recompute the prompt at each step. The context window is 512 tokens.

## Evaluation

Scores below come from zero-shot evaluation with lm-eval 0.4.12. BLiMP uses accuracy. ARC-Easy, ARC-Challenge, HellaSwag, and PIQA use length-normalized accuracy. WikiText-2 is reported as byte-level perplexity, where lower is better.

min-spark reaches 69.19% on BLiMP at medium effort. ARC-Easy reaches 37.08%. ARC-Challenge reaches 23.21%. HellaSwag reaches 27.92%. PIQA reaches 54.35%. Its best WikiText-2 byte perplexity is 2.7747.

### Effort levels

The table shows how the model responds to additional internal computation. Grammar improves most clearly from low to medium effort. The common-sense tasks remain close across the three settings.

![min-spark accuracy by effort](charts/effort.svg)

| Effort | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | WikiText-2 byte-ppl |
|--------|-------|----------|---------------|-----------|------|---------------------|
| min-spark-low | 67.11% | 35.10% | 23.21% | 27.91% | 54.13% | 2.8783 |
| min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
| min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |

### Comparison with small models

The comparison places min-spark alongside published results for GPT-S2-5M, SLM-10M, and michel-nano-v2. It covers the benchmarks reported across this group.

![min-spark compared with small-model peers](charts/comparison.svg)

| Model | Params | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA |
|-------|--------|-------|----------|---------------|-----------|------|
| **min-spark** | 5.76M | 69.19% | **37.08%** | 23.21% | **27.92%** | 54.35% |
| [GPT-S2-5M](https://huggingface.co/AxiomicLabs/GPT-S2-5M) | 5M | — | 33.92% | 22.87% | 27.87% | **57.56%** |
| [SLM-10M](https://huggingface.co/LiodonAI/SLM-10M) | 10M | — | 35.52% | **23.46%** | 27.40% | 57.07% |
| [michel-nano-v2](https://huggingface.co/finnianx/michel-nano-v2) | 8M | **72.52%** | 35.90% | 21.84% | 27.40% | 56.75% |

min-spark reaches 37.08% on ARC-Easy, the highest score in this comparison. Its 23.21% ARC-Challenge result is close to SLM-10M at 23.46%. HellaSwag reaches 27.92%, and PIQA reaches 54.35%. michel-nano-v2 records the highest BLiMP result in the group at 72.52%, while min-spark reaches 69.19%.

### Comparison with KeyLM

KeyLM is a 75.25M-parameter language model trained on 18B tokens. min-spark uses less than one thirteenth of its parameter count while reaching a similar range on several small-model evaluations.

KeyLM reports 29.9% on its ARC average, 29.7% on HellaSwag, and 60.0% on PIQA. min-spark reaches 37.08% on ARC-Easy, 23.21% on ARC-Challenge, 27.92% on HellaSwag, and 54.35% on PIQA. The comparison highlights the value of repeated computation in a compact model. min-spark closes much of the size gap on these tasks while retaining a 5.76M parameter footprint.

## Architecture

min-spark uses a tied-embedding, looped decoder with 5,758,572 parameters. Its 4,096-token byte-level BPE vocabulary feeds an embedding with model width 288. A prelude block prepares the hidden state, followed by a body of three distinct grouped-query attention blocks. The body is repeated according to the selected effort level before a coda block produces the final representation. The input embedding and output projection share weights.

Each pass includes loop-specific low-rank adaptation and a transient loop embedding. Deep-Delta residual projections provide the recurrent update. Across the full computation, the model applies 11 transformer blocks per token. The internal loop count is K=2 for low effort, K=3 for medium effort, and K=4 for high effort.

[![min-spark architecture](https://hfviewer.com/api/card.svg?source=MinimaLabs%2Fmin-spark&granularity=auto&animated=false)](https://hfviewer.com/MinimaLabs/min-spark)

Open the image to explore the full architecture graph in hfviewer, including the block-level structure and parameter flow.

## Training

min-spark was pretrained as a decoder-only base model on 10.01B tokens. The corpus combines filtered FineWeb-Edu with Finemath-4plus. Training used a warmup, stable learning-rate phase, and decay schedule with a 20% cooldown. The released checkpoint is the final checkpoint selected from this training run.

The model is released for research and experimentation with compact language models, weight-shared decoders, and controllable inference compute.

## Reproducing the evaluation

The evaluation runner is included in the repository. It supports each effort level and the benchmark suite used for this card.

```bash
python run_lmeval.py \
  --effort medium \
  --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext
```

Use `--limit N` to run a smaller evaluation during development.

## Limitations

min-spark is a small base language model. It is not instruction-tuned and does not provide conversational alignment or safety filtering. Factual recall, multi-step reasoning, and long-form coherence are limited by its scale and training objective.

The model supports a 512-token context window. Generation accepts one sequence at a time and currently has no KV cache. Results were collected with a 512-token context and should not be assumed to transfer to longer inputs.

## License

Apache-2.0. See [LICENSE](LICENSE).