---
language: en
license: mit
tags:
  - python
  - code-generation
  - mamba
  - attention
  - hybrid-architecture
  - small-language-model
  - program-synthesis
datasets:
  - bigcode/the-stack-dedup-v2
metrics:
  - perplexity
  - loss
model_name: abl_2_1_frontloaded
model_index:
  name: abl_2_1_frontloaded
---

# CRUMB `abl_2_1_frontloaded`

## Model Overview

`abl_2_1_frontloaded` is a hybrid decoder-only language model from the
**CRUMB** (Compact Recurrent-hybrid Underlying Mamba Blocks) project. It
interleaves **Mamba-3 selective state-space layers** with **GQA
(Grouped-Query Attention) layers** at a 2:1 Mamba-majority ratio with
**frontloaded** placement, and was pre-trained exclusively on Python source
code.

It is one of eleven ablation variants trained to study how the
Mamba-to-attention ratio and the placement of attention layers affect
small-scale (~150M-parameter) language models on Python program synthesis.

## Architecture

| Property | Value |
|---|---|
| Total parameters | **146,075,776** (146.1M) |
| `d_model` | 768 |
| `n_layers` | 12 |
| `n_heads` | 12 |
| `n_kv_heads` | 4 (GQA) |
| `d_head` | 64 |
| `d_ff` | 3072 |
| `vocab_size` | 32768 |
| `seq_len` | 4096 |
| Tie embeddings | yes |
| Pos. encoding | RoPE (base = 10000) |
| Mamba layer type | Mamba-3 (d_state=64, expand=2, headdim=64, ngroups=1, chunk=64) |

### Mamba : Attention ratio — **2 : 1**

8 Mamba layers + 4 GQA attention layers (2 Mamba blocks per attention
block).

### Placement — **Frontloaded**

Attention layers are stacked at the **beginning** of the network. Layer
order: `A A A A M M M M M M M M`

This placement forces attention-based token-level matching in the early
layers, with the final layers dominated by linear-time Mamba blocks before
the output projection.

## Training

| Property | Value |
|---|---|
| Training data | Python subset of `bigcode/the-stack-dedup-v2` |
| Tokens seen | **5,367,455,730** (~5.37 B) |
| Steps | 163,840 |
| Context length | 4096 |
| Training time | **47 h 44 m 12 s** |
| Final learning rate | 3.00e-05 |

## Evaluation Method

### Perplexity (primary metric)

Per-token cross-entropy loss with BF16 autocast, computed over the full
held-out evaluation set.

| Setting | Value |
|---|---|
| Eval sequences | 20,063 batches |
| Eval tokens | **328,631,940** |
| Implementation | `src/evaluation/perplexity.py` |

### Generation-based metrics

* **Python syntax validity** — 200 free-form completions generated per model
  from 49 diverse Python prompts at `temperature=0.8`, `top_k=50`,
  `max_new_tokens=128`; each completion checked with `ast.parse()`.
  Implementation: `src/evaluation/syntax_validity.py`.
* **Qualitative side-by-side completions** — 10 fixed prompts at
  `temperature=0.6`, `top_k=50`, `max_new_tokens=200`, identical random
  seed per prompt.
  Implementation: `src/evaluation/qualitative_comparison.py`.

## Evaluation Results

| Metric | Value |
|---|---|
| **Eval loss** | **1.3104** |
| **Eval perplexity** | **3.7078** |
| Eval time | 3,222.78 s (~54 min) |
| Syntax validity (n=200) | 67 / 200 → **33.5 %** ⭐ (highest) |
| Inference gen. time (200×128 tok) | 166.08 s |

## Rank Summary

Out of 11 ablation configurations evaluated at the same token budget:

| Rank | Model | Perplexity |
|---|---|---|
| 1 | `abl_2_1_interleaved` | 3.4182 |
| 2 | `abl_3_1_interleaved` | 3.4359 |
| 3 | `abl_3_1_backloaded` | 3.4493 |
| 4 | `abl_2_1_backloaded` | 3.4683 |
| 5 | `abl_1_1_backloaded` | 3.4763 |
| 6 | `abl_pure_mamba` | 3.5237 |
| 7 | `abl_1_1_interleaved` | 3.5407 |
| 8 | `abl_pure_attn` | 3.5939 |
| 9 | `abl_3_1_frontloaded` | 3.6798 |
| **10** | **`abl_2_1_frontloaded`** | **3.7078** |
| 11 | `abl_1_1_frontloaded` | 3.7315 |

`abl_2_1_frontloaded` ranks 10th overall. Frontloaded placement is
uniformly inferior across all three ratios — placing attention early forces
token-level matching before the network has built up enough
representational depth. Notably, `abl_2_1_frontloaded` has the **highest
syntax validity rate** of any model (33.5 %), but this is within the 95 %
confidence interval (±6.3 % at n=200) and is not statistically
discriminative.

## Intended Use & Limitations

* **Domain:** Python source-code language modelling.
* **Base model only:** no instruction tuning, no chat alignment, no safety
  filtering. Outputs are unconstrained code completions.
* **Repetitive degeneration:** all base CRUMB models tend to repeat
  function signatures / docstrings during free-form generation; this is
  expected behaviour for unaligned base models.

## Citation / Context

This model is part of the CRUMB Phase-1 ablation study:

> *Efficient Architectural Hybrids for Small-Scale Language Models in Python
> Program Synthesis* — Department of Computer Science and Engineering,
> Daffodil International University.
> Findings documented in `documents/phase1_ablation_findings.md`.

## How to Load

```python
from tokenizers import Tokenizer
import torch
from src.model.config import CRUMBConfig
from src.model.model import CRUMBModel

config = CRUMBConfig.from_yaml("configs/model/abl_2_1_frontloaded.yaml")
model = CRUMBModel(config)
state = torch.load("saved/model/abl_2_1_frontloaded/model.pt", map_location="cpu")
model.load_state_dict(state)
model.eval()

tok = Tokenizer.from_file("saved/tokenizer/crumb_tok_hf/tokenizer.json")
ids = tok.encode("def fibonacci(n):\n").ids
x = torch.tensor([ids])
with torch.no_grad():
    y = model(x)
```