---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- arithmetic
- math
- small-language-model
- bananamind
- digit-tokenizer
- pytorch
- safetensors
- custom-code
- trust-remote-code
- custom-architecture
model-index:
- name: MathBananaMind-1.1
  results:
  - task:
      type: multiple-choice
      name: Arithmetic multiple choice
    dataset:
      type: AxiomicLabs/ArithMark-2.0
      name: ArithMark 2.0
    metrics:
    - type: accuracy
      name: Accuracy
      value: 90.28
---

# MathBananaMind-1.1

MathBananaMind-1.1 is a tiny arithmetic specialist language model trained from scratch by BananaMind.

The model has **2,901,984 parameters** and reaches **90.28% on ArithMark 2.0** using the benchmark's official raw summed continuation log-likelihood scoring.

It was trained only on synthetic integer arithmetic. No general web, instruction, conversational, or other language-model datasets were used.

This is a base continuation model, not an instruction-tuned chat assistant.

## Model Details

| Field | Value |
|---|---:|
| Parameters | 2,901,984 |
| Architecture | Custom Llama-style decoder-only Transformer |
| Layers | 9 |
| Hidden size | 160 |
| Token embedding size | 32 |
| Intermediate size | 480 |
| Attention heads | 5 |
| KV heads | 1 |
| Head dimension | 32 |
| Attention style | Grouped-query attention |
| MLP | SwiGLU |
| Position embeddings | RoPE |
| RoPE theta | 10,000 |
| Normalization | RMSNorm |
| Vocabulary size | 8,192 |
| Context length | 1,024 |
| Token projection | Shared input/output token table with learned projections |
| Weight format | safetensors |
| Released checkpoint | Step 29,000 |
| Packed tokens at checkpoint | 950,272,000 |
| Model type | Base arithmetic causal LM |

The narrow 32-dimensional token table keeps compatibility with the 8k tokenizer while leaving most of the parameter budget for the Transformer blocks.

## Tokenizer

MathBananaMind-1.1 uses a custom 8k-token byte-level BPE tokenizer with digit-aware tokenization.

Every decimal digit is represented by its own token:

| Token | ID |
|---|---:|
| `1` | 9 |
| `2` | 10 |
| `3` | 11 |
| `4` | 12 |
| `5` | 13 |
| `6` | 14 |
| `7` | 15 |
| `8` | 16 |
| `9` | 17 |
| `0` | 18 |

Special token IDs:

| Token | ID |
|---|---:|
| `<|endoftext|>` | 0 |
| `<|pad|>` | 1 |
| `<|unk|>` | 2 |


## Demo
You can try out the model using this Demo: https://huggingface.co/spaces/hugging-apps/mathbananamind-1-1
## Training Data

The model was trained exclusively on synthetic integer equations in this exact ASCII continuation format:

```text
59 + 45 = 104
(16 / 4) + 44 = 48
3 * 9 + 12 / 1 = 39
```

The corpus covers:

- Addition, subtraction, multiplication, and exact division
- One, two, and three operators
- Standard operator precedence
- Parenthesized two- and three-operator expressions
- Zero operands
- Boundary and high-carry examples
- One- to three-digit operands in the ArithMark-focused release mix

| Field | Value |
|---|---:|
| Unique generated examples | 644,655 |
| Unique corpus tokens | 10,000,014 |
| Full-run packed tokens | 1,000,013,824 |
| Full-run supervised answer tokens | 246,037,802 |
| Other datasets used | 0 |

The corpus was streamed repeatedly with a 50% one-operator, 30% two-operator, and 20% three-operator training mix.

Training used answer-only supervision. Context and EOS labels were masked, so the objective directly optimized the continuation tokens scored by ArithMark.

### Contamination Guard

The training distribution and surface form were deliberately aligned to ArithMark 2.0. To prevent exact benchmark leakage, all 2,500 normalized ArithMark contexts were loaded before generation and hard-excluded.

The generator dropped 135,536 candidate collisions, deduplicated normalized expressions on disk, and asserted zero exact context leakage before writing the final manifest.

## Training Setup

| Field | Value |
|---|---:|
| Sequence length | 256 |
| Micro batch | 32 |
| Gradient accumulation | 4 |
| Tokens per optimizer step | 32,768 |
| Full-run optimizer steps | 30,518 |
| Released checkpoint step | 29,000 |
| Optimizer | AdamW |
| Betas | 0.9, 0.95 |
| Peak learning rate | 1e-3 |
| Warmup steps | 100 |
| LR schedule | Linear warmup with cosine decay |
| Minimum LR ratio | 0.3 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Seed | 42 |

## Evaluation

Self-reported evaluation on all 2,500 examples from `AxiomicLabs/ArithMark-2.0`.

The published score uses raw summed log-likelihood over each candidate continuation, matching the benchmark implementation. The release checkpoint was evaluated in float32.

### Benchmark Summary

| Model | Parameters | ArithMark 2.0 |
|---|---:|---:|
| **MathBananaMind-1.1** | **2.90M** | **90.28%** |

### Score by Operator Count

| Operator count | Correct | Total | Accuracy |
|---:|---:|---:|---:|
| 1 | 1,139 | 1,250 | 91.12% |
| 2 | 698 | 750 | 93.07% |
| 3 | 420 | 500 | 84.00% |

### Single-Operator Scores

| Operator | Topic | Correct | Total | Accuracy |
|---|---|---:|---:|---:|
| `+` | Addition | 523 | 538 | 97.21% |
| `-` | Subtraction | 411 | 438 | 93.84% |
| `*` | Multiplication | 105 | 144 | 72.92% |
| `/` | Division | 100 | 130 | 76.92% |

### Accuracy on Every Example Containing Each Operator

These groups overlap because a mixed expression can contain multiple operators.

| Operator | Correct | Total | Accuracy |
|---|---:|---:|---:|
| `+` | 1,393 | 1,514 | 92.01% |
| `-` | 1,100 | 1,208 | 91.06% |
| `*` | 745 | 902 | 82.59% |
| `/` | 438 | 487 | 89.94% |

### Score by Topic

| Topic | Accuracy |
|---|---:|
| Addition | 97.21% |
| Subtraction | 93.84% |
| Multiplication | 72.92% |
| Division | 76.92% |
| Mixed two operators | 95.19% |
| Parentheses with two operators | 90.70% |
| Mixed three operators | 88.84% |
| Parentheses with three operators | 79.46% |

## Repository Files

| File | Description |
|---|---|
| `config.json` | Transformers configuration for `mathbananamind` |
| `model.safetensors` | Step-29,000 model weights |
| `tokenizer.json` | Custom 8k digit-aware tokenizer |
| `tokenizer_config.json` | Tokenizer metadata |
| `generation_config.json` | Deterministic generation defaults |
| `configuration_mathbananamind.py` | Custom Transformers configuration class |
| `modeling_mathbananamind.py` | Custom Transformers causal LM implementation |
| `evaluation_results.json` | Machine-readable benchmark summary |

## Usage

This model uses custom architecture code, so load it with `trust_remote_code=True`.

Install dependencies:

```bash
pip install -U transformers safetensors torch
```

Run a direct arithmetic continuation:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/MathBananaMind-1.1"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

prompt = "(16 / 4) + 44 ="
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=3,
        do_sample=False,
        use_cache=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

continuation = tokenizer.decode(
    output[0, inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)
print(prompt + continuation)
```

The model was trained for direct continuations such as `expression = answer`. It was not trained to follow chat instructions or produce chain-of-thought explanations.

## Intended Use

MathBananaMind-1.1 is intended for small-model arithmetic research, continuation-likelihood evaluation, digit-tokenizer experiments, and lightweight local inference.

## License

Apache 2.0

## Citation

```bibtex
@misc{mathbananamind11,
  title = {MathBananaMind-1.1},
  author = {BananaMind},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/BananaMind/MathBananaMind-1.1}}
}
```