oxfrug/Ornith-1.5-35B-A3B-REAP-20pct

Ornith-1.5-35B-A3B REAP 20%

20% expert-pruned variant of ornith-ai/Ornith-1.5-35B-A3B using REAP (Lasby et al., Cerebras, ICLR 2026).

Note: This is not a lossless compression. The base architecture is a Qwen3.5 hybrid (GatedDeltaNet + MoE). Using the same pruning recipe on 0xSero's Qwen3.5-35B yielded a ~3-point drop on HumanEval and raised WikiText perplexity from 6.83 → 9.51. Expect a comparable performance tax here. No official SWE-bench or Terminal-Bench scores are claimed.

What changed

	Base	This
Experts / layer	256	205 (51 removed)
Active / token	8	8
Layers	40	40
On-disk BF16	~67 G	~56 G

Shared expert, vision tower, embeddings, and lm_head are untouched. Router weights renormalized after prune.

Cal

Same mix as 0xSero/Qwen3.5-28B, not Cerebras’ 24k×16k agentic mix:

• theblackcat102/evol-codealpaca-v1 × 250
• open-r1/Mixture-of-Thoughts code / math / science × 250 each
• 1000 sequences, max length 1024 (3090 VRAM; 0xSero used 4096)
• seed 42, prune_method=reap, renormalize_router_weights=true
• Layerwise observer (model does not fit in 60 G RAM)

No xlam / SWE-smith traces — extra risk for Ornith’s tool-calling specialists.

Convert / load

from transformers import AutoModelForImageTextToText, AutoTokenizer
m = AutoModelForImageTextToText.from_pretrained(
    "oxfrug/Ornith-1.5-35B-A3B-REAP-20pct", torch_dtype="auto", device_map="auto"
)

config.json text_config.num_experts is 205. Needs a Qwen3.5-MoE stack (transformers with Qwen3_5MoeForConditionalGeneration).

Smoke

Greedy, thinking off, 3090 offload. Not a published coding bench.

• square lambda / is_even — match base
• get_weather with the base card tools= schema — well-formed <tool_call><function=get_weather> / Paris on both base and this prune

License

MIT, same as the base. Prune by oxfrug. Method: Lasby et al., REAP the Experts, ICLR 2026.