texassred/LFM2.5-8B-A1B-heretic-16e-GGUF

LFM2.5-8B-A1B-heretic 16e — the expert-cut, gate-schooled coffin brain

A leaner, fully-resident reincarnation of coder3101/LFM2.5-8B-A1B-heretic: the 32 routed experts per layer were cut to 16, and the router gates were retrained (teacher-prior school) so the surviving heads know where to route. Same soul, half the mass — built to live entirely inside a 4GB VRAM card with real context headroom.

Why this exists

The full 8B model is 4.0G at IQ4_XS — it technically fits a 4GB card but leaves almost no room for context. Cutting experts cuts the file, not the brain per token: top-4 stays top-4, so the active parameters (A1B) are identical to the original — same speed, half the weight.

Model details

Property	Value
Architecture	lfm2_moe (LFM2.5, linear attention)
Layers	24 (2 dense + 22 sparse MoE floors)
Experts per sparse floor	16 (was 32)
Experts per token	4 (top-4)
Active params / token	~1B (A1B, unchanged from base)
Total params	4.6B
Context length	128,000 native (linear attention = tiny KV)
Hidden size	2048
MoE intermediate	1792
Quantization	IQ4_XS with imatrix (this file)
Base	coder3101/LFM2.5-8B-A1B-heretic

What was done

1. Expert cut (32 → 16): per-layer expert tensors (w1/w2/w3) sliced to the first 16, gate rows and expert bias sliced to match, num_experts updated in config.
2. Gate school (teacher-prior proxy): the ORIGINAL 32-expert gates were kept as teacher. Per-token soft targets = softmax of the original routing preference over the surviving 16 experts, blended with a uniform prior. Only the 16-wide gate weights were trained (CE, AdamW) — no backprop through the body. The cut brain learns where to route using its own old wisdom.
3. Quant: F16 → imatrix (custom calibration corpus) → IQ4_XS.

The full bf16 safetensors (model.safetensors) ships in this repo — the gates are already patched in, so this is a complete, loadable model, not a patchset.

Files

File	Description
model.safetensors	Full bf16 weights, post-trained gates (8.6G)
lfm25-heretic-16e-iq4-xs.gguf	IQ4_XS quant, imatrix-calibrated (2.4G)
config.json / tokenizer*	Transformers-compatible config + tokenizer

Measured

• Generation: ~29.7 tok/s on i5-12450H CPU (8 threads, -c 2048) — 4x faster than the 24-expert cut.
• Prompt eval: ~58 tok/s.
• Coherent chat output after school (no word salad; verified via llama-server and ollama).
• Fits 4GB VRAM at IQ4_XS with ~1.5G left for context. Linear attention keeps KV tiny, so long context is cheap.

Usage

Ollama (import from this GGUF — a tuned Modelfile ships in this repo):

ollama create lfm25-heretic-16e -f Modelfile.lfm25-heretic-16e

(Tuned sampling: temperature 0.8, top_p 0.95, top_k 60, repeat_penalty 1.15. Note: this is a thinking model — ollama run shows <think> blocks before the answer.)

llama.cpp (RECOMMENDED for harness/API use — clean reasoning split):

llama-server -m lfm25-heretic-16e-iq4-xs.gguf --jinja --reasoning-format deepseek -c 8192

The GGUF carries the official LFM2.5 chat template (with the bos_token literal fix — llama.cpp doesn't pass bos_token as a template var, so the raw official template renders a 4-token prompt; <|startoftext|> is baked in). Responses arrive with the thinking in message.reasoning_content and the answer clean in message.content — verified: {"answer": "2+2 = 4", "reasoning_content": "..."}.

llama.cpp:

llama-server -m lfm25-heretic-16e-iq4-xs.gguf --jinja -c 8192
# or
llama-cli -m lfm25-heretic-16e-iq4-xs.gguf -p "Hi! Who are you?"

Transformers (from the safetensors):

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("texassred/LFM2.5-8B-A1B-heretic-16e-GGUF", trust_remote_code=True)

Notes

• Small-active brain (~A1B per token): excellent for chat, explanation, code-adjacent work; arithmetic is not its superpower (a 1B active brain ceiling, not a defect of the cut).
• The base model's "heretic" ablation carries through — this is an uncensored model. Use accordingly.
• Built with the forge scripts: expert slice, teacher-prior gate school, imatrix IQ4_XS pipeline. The same pipeline makes any future cut (12e, 8e, ...) in about an hour.

Credits

Base: coder3101/LFM2.5-8B-A1B-heretic. Cut + school + quant: texassred (Sekhmet forge).