Escarda-86M-Base

Escarda-86M-Base is a ~86M-parameter, from-scratch decoder-only language model — the base sibling of Quazim0t0/Escarda-86M (the chat-tuned model). It shares the same SpikeWhaleLM architecture (Multi-head Latent Attention, an n-gram "engram" memory, hash-lookup layers, hyper-connections, an HRM refinement step, and JEPA / multi-token-prediction training objectives) and the same custom ChatML-aware tokenizer.

This checkpoint is a JEPA-distilled base. It is best used as a starting point for continued pretraining / fine-tuning rather than as a chat assistant.

Related models: SFT / chat model → Quazim0t0/Escarda-86M · live demo → Escarda-86M-Chat Space

Trained using Modal's credits during the Small Models, Big Adventures Hackathon.

---

Model summary

Parameters	~85.7M (tie_word_embeddings=True)
Type	Decoder-only LM (SpikeWhaleLM, model_type: spike_whale)
Hidden size / layers	640 / 16
Attention	10 heads (head_dim=64), 1 KV head (MQA), MLA low-rank Q/O, decoupled RoPE(16)+NoPE(48), QK-norm
Context length	4096 tokens
Vocab	16,512 (custom length-max tokenizer)
License	Apache-2.0

For the full architecture description see the chat model's card.

---

Architecture

These models are built on SpikeWhaleLM, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components:

• Multi-head Latent Attention (MLA + XSA) — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a single KV head (multi-query attention), with QK-norm for stable logits.
• Engram n-gram memory — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream.
• Hash-lookup layers (×2) — multi-head content-addressable features alongside the token embeddings.
• Hyper-Connections — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add.
• HRM refinement — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head.
• Multi-Token Prediction (MTP) — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost).
• Feed-forward is dense (the block is MoE-capable, but MoE is disabled in this release).

JEPA vs HRM. The Escarda models are trained with both HRM refinement and a JEPA (Joint-Embedding Predictive Architecture) auxiliary objective (use_hrm_refine=True, use_jepa=True) — the JEPA term predicts future latent states during training to shape the model's representations. The sibling Byrne models drop JEPA and use HRM refinement only.

Tokenizer

These models use SpikeTokenizer, a custom byte-level "length-max" (greedy longest-match) tokenizer with a 16,512-token vocabulary — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is ChatML-aware, with atomic special tokens for framing and reasoning/tool markers (<|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers) plus <bos>/<eos>/<pad>/<unk>. It ships as a PreTrainedTokenizer subclass (spike_tokenizer.py) and loads via AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

splits. byte_ppl is exp(sum_NLL_nats / total_UTF8_bytes) on WikiText-2 test (tokenizer- independent). BLiMP is fraction of minimal pairs with logprob(good) > logprob(bad) (12 paradigms × 150). Stderr is binomial sqrt(p(1-p)/n).

Language modeling

Metric	Value
WikiText-2 byte_ppl ↓	2.2228
BLiMP acc ↑	0.7144

Multiple-choice suite

Task	acc	±	acc_norm	±
arc_easy	0.3801	0.0100	0.3615	0.0099
arc_challenge	0.1886	0.0114	0.2235	0.0122
hellaswag	0.2759	0.0045	0.2832	0.0045
winogrande	0.5162	0.0140	—	—
piqa	0.5843	0.0115	0.5631	0.0116
openbookqa	0.1300	0.0150	0.2500	0.0194
boolq	0.5138	0.0087	—	—

ArithMark-2.0 (AxiomicLabs)

Metric	Value
acc	0.2536 ± 0.0087
acc_norm	0.2348 ± 0.0085

n = 2,500 · chance = 0.25.

Note: as a distilled base, this checkpoint has the lowest byte-perplexity of the Escarda family but trades off downstream task accuracy — a good reminder that perplexity alone is not a reliable capability ranking. For the strongest chat behaviour use Escarda-86M; use this model when you want a low-loss base to continue pretraining or fine-tune.

---

Training & token budget

• Tokens: ~20B (from-scratch pretraining of the SpikeWhale base, ~28k steps); this checkpoint is a JEPA-distilled snapshot of that base.
• Token/param ratio: ~233 tokens/param (20B / 85.7M) — roughly 11–12× the Chinchilla ~20-tokens/param compute-optimal heuristic, i.e. a deliberately over-trained small model (the inference-efficient trade-off).

Fitting the Chinchilla data term to this model's own pretraining loss curve gives:

L(D) ≈ 2.611 + 77,715 · D^(−0.537) (nats/token, R² = 0.92)

From that fit:

• Compute-optimal tokens for this 86M size ≈ 4.3B → the 20B run is ~4.6× past compute-optimal.
• Diminishing-returns knee ≈ 22.5B tokens (where +1B tokens buys < 0.005 nats) — the 20B stopping point lands right at the knee, a well-judged budget.
• The model is parameter-bound, not data-bound at 20B: the capacity term (0.82 nats) exceeds the data term (0.54), so extra tokens help little. Doubling to 40B is projected to lower loss only 0.07 nats (7% perplexity) with negligible downstream gain — the lever for better quality is more parameters, not more tokens. (This is also why, as a distilled base, it reaches the lowest perplexity of the family without the best downstream scores — it is already at its data-term floor.)

Caveats: single-size fit (folds irreducible loss + capacity floor into one constant); the cosine-LR decay inflates the fitted exponent, so treat β as an upper bound; token counts are anchored to the ~20B figure and scale linearly if that differs.

---

Usage

Custom architecture — load with trust_remote_code=True (the modeling code ships in this repo via auto_map):

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Quazim0t0/Escarda-86M-Base", trust_remote_code=True)

The tokenizer is the custom SpikeTokenizer (tokenizer.json, algorithm: length-max); load it with the spike_tokenizer.py helper from the project rather than AutoTokenizer.

Acknowledgements

Built with Modal credits during the Small Models, Big Adventures Hackathon, and released to the community as a base to build on.