SPROG-9M: how far a 9-million-parameter, LLM-free model gets on grade-school math

We trained a tiny model to solve math word problems without using any large language model at inference, and wrote up what actually worked.

The setup

GSM8K is a benchmark of about 8,500 grade-school math word problems. They read like this: "A baker had 24 muffins, sold three-quarters of them, then baked 10 more. How many does she have now?" A modern LLM scores above 90% on them. They are easy for a person and easy for a big model.

We were after a different question. How far can you get with no large language model at all, under three strict rules:

1. At most 10 million trainable parameters. A small LLM has one to eight billion. We allowed ourselves roughly a thousand times fewer.
2. From scratch. No pretrained weights, and no frozen foundation model with a small probe bolted on. The model begins as random numbers.
3. No looking at the test set during development.

The result is SPROG-9M, a 9.37M-parameter model that reaches about 11.8% on the held-out GSM8K test set. That is a small number in absolute terms, but for a from-scratch model with no LLM it is a real climb from the roughly 1% a naive approach gets. The number matters less than what we learned getting there, including several results that looked strong until we measured them properly.

Both the model and the training data are public:

• Model: https://huggingface.co/codelion/sprog-9m
• Dataset: https://huggingface.co/datasets/codelion/gsm8k-synth

The core idea: predict a program, not an answer

A tiny model has no business learning arithmetic from raw text, so we do not ask it to. We split the task into two parts and let the model handle only the part it can.

First, we abstract the numbers into slots. Every number in the question is replaced by a placeholder token in order of appearance: [N0], [N1], [N2], and so on. Spelled-out numbers like "twenty-four" are converted to digits first. The muffin problem becomes:

"A baker had [N0] muffins, sold [N1]/[N2] of them, then baked [N3] more. How many now?"

The model never sees 24 or 10. It only sees structure. This is the central trick. It forces the model to learn how problems are shaped rather than memorizing particular numbers. That structure generalizes across every problem with the same shape, regardless of the actual values.

Second, the model predicts a small program over those slots, written in postfix (reverse-Polish) notation:

[N0] [N0] [N1] [N2] / * -

which means "N0 minus (N0 times N1 divided by N2)". A 30-line deterministic executor substitutes the real numbers back in (24 minus 24 times 3 divided by 4 leaves 6, then plus 10 baked gives 16) and computes the result. Python does the arithmetic exactly, every time. The model only has to get the shape of the computation right.

We also give it a small table of constant tokens, such as [C100] for percentages or [C0.5] for "half", to cover values that appear in the math but not in the text.

The idea of predicting an expression and executing it goes back to the pre-LLM math-word-problem literature. We took it to a deliberately tiny model.

The model

SPROG is a compact encoder-decoder transformer built in MLX, Apple's array framework, so it runs on a laptop. The configuration:

• width 304, with 4 encoder layers and 4 decoder layers, 4 attention heads
• a 6,000-token source vocabulary, built only from the training questions
• a 79-token target vocabulary: the operators, the 20 slots, and the constants
• 9.37M parameters in total, most of it sitting in the embedding tables

There is nothing exotic here. We did try fancier designs, including a graph encoder over the quantities, a recurrent Universal-Transformer-style model, and auxiliary structure-prediction heads. None of them beat the plain seq2seq once we measured carefully. Capacity was never the limiting factor.

The free verifier, where a third of the accuracy comes from

A single greedy decode is right about 8 to 9% of the time. The jump to roughly 12% comes from a step that adds zero parameters.

At inference we draw 96 samples from the model at temperature 0.9, which gives 96 candidate programs. Each executes to some answer. Then a symbolic verifier picks one. The verifier never sees the correct answer. It scores each candidate on internal sanity alone:

• Coverage: does the program use more of the numbers in the problem? GSM8K rarely has distractor numbers, so using them is a good sign.
• Magnitude: is the answer in a plausible range, or did the program blow up?
• Intermediate values: are all the intermediate results non-negative and whole? You cannot have minus three apples or two and a half children.
• Vote frequency, used only to break ties between candidates that are otherwise equally sane.

The verifier is a few dozen lines of rules with no training and no parameters. It raises the most-voted-answer accuracy from about 9.3% to about 11.8%, a gain of 2.5 points at no cost to the parameter budget. No other component does as much, and it never touches the 10M cap.

A fair question: is 96 samples just pass@96? No. Pass@k asks whether the correct answer appears anywhere in the k samples, which means it checks against the answer key, so it is an oracle and an upper bound. For us that figure is about 39%. Our 11.8% is the verifier committing to a single answer with no answer key available. It belongs to the self-consistency family of metrics, well below the oracle.

Why 96 and not more? We measured it. Recall keeps rising with sample count: the gold answer is in the pool about 39% of the time at 96 samples and about 50% at 288. But the verifier's ability to convert that recall peaks around 64 to 96 samples and then falls. Past that point the extra samples mostly add plausible but wrong distractors that confuse the selection (192 samples scored 8.5%, 288 scored 8.3%). 96 sits at the sweet spot.

The real lever: data

This was the most surprising part of the project. We put a lot of effort into architecture and selection, and they barely mattered. Almost all of the progress came from the training data.

A from-scratch 9M model needs to see many shapes of problem to generalize, and GSM8K's roughly 7,500 training problems are not enough. So we generated more, using LLMs as data authors. The solver itself stays free of any LLM at inference; the models only write training examples.

We prompted Claude and Gemini to write new GSM8K-style problems across many domains (stores, farms, bakeries, sports, and so on) and many reasoning patterns (rates, percentages, comparisons, money, fractions, running totals). Each generated problem includes a fully worked solution with inline annotations like <<3*4=12>>, which lets us validate it automatically. We extract the program from the annotations, re-run it, and keep the problem only if it reproduces the stated answer. About 85% survive.

That gave us 117,955 validated problems, 20,676 from Claude and 97,279 from Gemini, released as the gsm8k-synth dataset.

Two parts of the data work were not optional.

Decontamination came first. An LLM can sometimes regenerate a near-copy of a real test problem. So we checked every generated problem against the GSM8K test set using 8-gram overlap, and removed anything with 50% or more overlap. The result was 0% contamination. At one point a library bug silently skipped this check and reported a false clean. We caught it, changed the loader to fail loudly instead of skipping quietly, and re-verified.

Quality filtering came second. Gemini's data had a specific flaw. About 1.2% of the time it would apply a fraction or percentage to a count of discrete objects. The result was something like "12.5 party hats", sometimes with a ramble about it in the solution. These pass the arithmetic check but teach the model nonsense. So we built a filter to catch and drop them. Claude's data had this defect 19 times less often.

What looked true on dev and was not

This section is the most useful thing we can pass along, because the mistake behind it is easy to make.

During development we evaluated on a held-out slice of the training set: 800 problems we never trained on, which we call the dev set. Saving the real test set for the very end is the responsible thing to do. But 800 problems at around 12% accuracy carries a noise band of roughly plus or minus 1.5 points, and that noise fooled us more than once.

Three separate wins looked solid on dev and then vanished when we finally ran the 1,319-problem test set.

The first was "deeper data is better". We improved the Gemini prompt to produce deeper, more multi-step problems, raising the average number of reasoning steps from 3.3 to 4.0. On a controlled dev comparison it won by a clean, non-overlapping 0.94 points. On the real test set the supposedly better data was worse, because it overshot the test distribution. GSM8K problems average about 3.7 steps, our deeper data was harder than the test, and training on harder-than-the-test data hurt. What mattered was matching the real distribution, not maximizing depth.

The second was "a bigger, deeper model helps". At a larger data scale, a 9.4M deep model beat the 5.6M model by over a point on dev. On test, across three random seeds, it landed inside the noise of the smaller model.

The third was "high seed variance means the model is unstable". On the 800-row dev set, three identical training runs scored 14.5%, 12.1%, and 13.5%, a frightening 2.4-point spread. On the 1,319-row test set the same three models scored 11.7%, 11.8%, and 12.1%, a 0.4-point spread. The instability was dev-set sampling noise all along. The model was steady; the small dev set made it look otherwise.

The takeaway is simple. A small dev set will sell you results that do not exist. Evaluate on enough data, report the mean across seeds, and treat anything inside the noise band as a tie. Most of our exciting intermediate numbers were noise. The data-quality and decontamination work was the part that held up.

Results

On the full GSM8K test set (1,319 problems), averaged over three training seeds:

For context, a fine-tuned GPT-3 with 175 billion parameters scored about 33% on GSM8K when the benchmark came out in 2021. SPROG reaches about a third of that with roughly 19,000 times fewer parameters, no pretraining, and no LLM at inference. It runs on a laptop CPU.

The wall

The gap between the 39% oracle and the 11.8% we convert is the whole story of what is left. The correct answer sits in the candidate pool 39% of the time, but no verifier we built, including learned ones, converts more than about 12%. The reason is that the wrong candidates are not random noise. They are plausible siblings of the right answer, off by a single operation, and telling them apart requires actually understanding the problem. At 9M parameters trained on 7K real examples, that understanding is not there. Selection turns out to be as hard as solving.

So about 12% is a real ceiling for this recipe. Getting meaningfully past it would need a much larger model and data regime, or dropping the no-LLM rule, at which point the interesting tiny-model experiment is gone.

Takeaways

• Split the labor. Let a tiny model predict structure, a program over number-slots, and let a deterministic executor do the arithmetic. Do not make a small model learn math.
• Free verifiers are underrated. A zero-parameter symbolic checker over a handful of samples gave us a third of our accuracy with no cost to the parameter budget.
• Data quality beat everything else. Model size, architecture, and clever selection all came out within noise. Matching the real distribution and decontaminating carefully were what moved the test number.
• Trust the test set and distrust a small dev set. Three of our best-looking results were dev-set mirages, and only multi-seed test evaluation told us so.

The model and the 117K-problem dataset are open. The interesting part is the path. A model you could train on a laptop, with no LLM and no pretraining, can reach this at all. Getting there meant throwing out a pile of results that turned out to be noise.

SPROG stands for Symbolic PROGram solver. Built and evaluated in MLX on Apple Silicon.