---
language: en
license: mit
tags:
  - python
  - code-generation
  - mamba
  - attention
  - hybrid-architecture
  - small-language-model
  - program-synthesis
datasets:
  - bigcode/the-stack-dedup-v2
metrics:
  - perplexity
  - loss
model_name: abl_3_1_backloaded
model_index:
  name: abl_3_1_backloaded
---

# CRUMB `abl_3_1_backloaded`

## Model Overview

`abl_3_1_backloaded` is a hybrid decoder-only language model from the
**CRUMB** (Compact Recurrent-hybrid Underlying Mamba Blocks) project. It
interleaves **Mamba-3 selective state-space layers** with **GQA
(Grouped-Query Attention) layers** at a 3:1 Mamba-majority ratio with
**backloaded** placement, and was pre-trained exclusively on Python source
code.

It is one of eleven ablation variants trained to study how the
Mamba-to-attention ratio and the placement of attention layers affect
small-scale (~150M-parameter) language models on Python program synthesis.

## Architecture

| Property | Value |
|---|---|
| Total parameters | **148,210,992** (148.2M) |
| `d_model` | 768 |
| `n_layers` | 12 |
| `n_heads` | 12 |
| `n_kv_heads` | 4 (GQA) |
| `d_head` | 64 |
| `d_ff` | 3072 |
| `vocab_size` | 32768 |
| `seq_len` | 4096 |
| Tie embeddings | yes |
| Pos. encoding | RoPE (base = 10000) |
| Mamba layer type | Mamba-3 (d_state=64, expand=2, headdim=64, ngroups=1, chunk=64) |

### Mamba : Attention ratio — **3 : 1**

9 Mamba layers + 3 GQA attention layers (3 Mamba blocks per attention
block).

### Placement — **Backloaded**

Attention layers are stacked at the **end** of the network. Layer order:
`M M M M M M M M M A A A`

This placement keeps the final layers — closest to the output projection —
precise and attention-based, while the earlier layers compress the input
through linear-time Mamba blocks.

## Training

| Property | Value |
|---|---|
| Training data | Python subset of `bigcode/the-stack-dedup-v2` |
| Tokens seen | **5,367,406,590** (~5.37 B) |
| Steps | 163,840 |
| Context length | 4096 |
| Training time | **47 h 43 m 14 s** |
| Final learning rate | 3.00e-05 |

## Evaluation Method

### Perplexity (primary metric)

Per-token cross-entropy loss with BF16 autocast, computed over the full
held-out evaluation set.

| Setting | Value |
|---|---|
| Eval sequences | 20,063 batches |
| Eval tokens | **328,631,940** |
| Implementation | `src/evaluation/perplexity.py` |

### Generation-based metrics

* **Python syntax validity** — 200 free-form completions generated per model
  from 49 diverse Python prompts at `temperature=0.8`, `top_k=50`,
  `max_new_tokens=128`; each completion checked with `ast.parse()`.
  Implementation: `src/evaluation/syntax_validity.py`.
* **Qualitative side-by-side completions** — 10 fixed prompts at
  `temperature=0.6`, `top_k=50`, `max_new_tokens=200`, identical random
  seed per prompt.
  Implementation: `src/evaluation/qualitative_comparison.py`.

## Evaluation Results

| Metric | Value |
|---|---|
| **Eval loss** | **1.2382** |
| **Eval perplexity** | **3.4493** |
| Eval time | 3,102.31 s (~52 min) |
| Syntax validity (n=200) | 63 / 200 → **31.5 %** |
| Inference gen. time (200×128 tok) | 184.11 s |

## Rank Summary

Out of 11 ablation configurations evaluated at the same token budget:

| Rank | Model | Perplexity |
|---|---|---|
| 1 | `abl_2_1_interleaved` | 3.4182 |
| 2 | `abl_3_1_interleaved` | 3.4359 |
| **3** | **`abl_3_1_backloaded`** | **3.4493** |
| 4 | `abl_2_1_backloaded` | 3.4683 |
| 5 | `abl_1_1_backloaded` | 3.4763 |
| 6 | `abl_pure_mamba` | 3.5237 |
| 7 | `abl_1_1_interleaved` | 3.5407 |
| 8 | `abl_pure_attn` | 3.5939 |
| 9 | `abl_3_1_frontloaded` | 3.6798 |
| 10 | `abl_2_1_frontloaded` | 3.7078 |
| 11 | `abl_1_1_frontloaded` | 3.7315 |

`abl_3_1_backloaded` is the **best backloaded variant across all three
ratios** (PPL 3.449) and the **3rd-best model overall**, narrowly behind
the two interleaved leaders. This result supports the broader finding that
backloaded attention is competitive with interleaved attention, and that
Mamba-majority hybrids (3:1) are competitive with — but not superior to —
2:1 hybrids at this scale.

## Intended Use & Limitations

* **Domain:** Python source-code language modelling.
* **Base model only:** no instruction tuning, no chat alignment, no safety
  filtering. Outputs are unconstrained code completions.
* **Repetitive degeneration:** all base CRUMB models tend to repeat
  function signatures / docstrings during free-form generation; this is
  expected behaviour for unaligned base models.

## Citation / Context

This model is part of the CRUMB Phase-1 ablation study:

> *Efficient Architectural Hybrids for Small-Scale Language Models in Python
> Program Synthesis* — Department of Computer Science and Engineering,
> Daffodil International University.
> Findings documented in `documents/phase1_ablation_findings.md`.

## How to Load

```python
from tokenizers import Tokenizer
import torch
from src.model.config import CRUMBConfig
from src.model.model import CRUMBModel

config = CRUMBConfig.from_yaml("configs/model/abl_3_1_backloaded.yaml")
model = CRUMBModel(config)
state = torch.load("saved/model/abl_3_1_backloaded/model.pt", map_location="cpu")
model.load_state_dict(state)
model.eval()

tok = Tokenizer.from_file("saved/tokenizer/crumb_tok_hf/tokenizer.json")
ids = tok.encode("def fibonacci(n):\n").ids
x = torch.tensor([ids])
with torch.no_grad():
    y = model(x)
```