OpenMOSE/Qwen3.5-REAP-262B-A17B-GGUF

OpenMOSE/Qwen3.5-REAP-262B-A17B-GGUF

Vision–Language MoE model created by applying Router-weighted Expert Activation Pruning (REAP) to Qwen3.5-397B-A17B.

---

1. Model Summary

• Base model: Qwen/Qwen3.5-397B-A17B (vision–language MoE LLM)
• Variant name: Qwen3.5-REAP-262B-A17B
• Architecture: Decoder-only Transformer + MoE MLP experts, with vision encoder + VL fusion as in Qwen3.5
• Pruning method: REAP (Router-weighted Expert Activation Pruning) by Cerebras Research, 3072 samples https://github.com/CerebrasResearch/reap
• Expert sparsity: ~35% of MoE experts pruned globally (512 → 333 experts)
• Active parameters: "A17B" indicates roughly ~17B active parameters per token (MoE sparse activation), while total parameters are reduced to about 262B
• Modality: Text + Vision (VL support kept intact)
• License: Apache 2.0
• Author / Maintainer: OpenMOSE
• Year: 2025

This is an unofficial community variant of Qwen3.5, not affiliated with or endorsed by Alibaba or Cerebras Systems.

---

2. What Is REAP and What Did We Change?

REAP (Router-weighted Expert Activation Pruning) is a pruning method for MoE models that uses:

• Router statistics (routing probabilities)
• Expert activation patterns on a calibration set

to identify under-used or redundant experts and prune them while preserving model quality as much as possible.

For this model:

• We applied REAP to Qwen3.5-397B-A17B across its MoE MLP blocks.
• ~35% of experts are pruned (512 → 333), based on router-weighted activation statistics.
• The routing mechanism itself is not conceptually changed; we only changed which experts remain.
• We extended the original REAP implementation to support the Qwen3.5 architecture, including its hybrid linear/full attention layers and vision components, so pruning can be applied without breaking VL functionality.

In short: same REAP algorithm, adapted to Qwen3.5, leaving VL functionality available.

---

3. Calibration Data

The REAP pruning statistics were computed using:

• Calibration dataset: https://huggingface.co/datasets/OpenMOSE/reap-calib-mix

• This dataset is mostly synthetic, generated by Qwen3-235B-Instruct on mixed prompts designed to cover:

  • General instruction-following
  • Reasoning and long-form text

The calibration set is not used for additional fine-tuning; it is used only to measure router/expert activations to decide which experts to prune.

---

4. Why 262B-A17B? (Motivation & Hardware Footprint)

By pruning ~35% of experts while keeping VL:

• The model shrinks from ~397B total parameters to about 262B total parameters.
• With sparse MoE activation, around 17B parameters are active per token ("A17B").
• In practice, this makes it feasible to deploy on a single 96 GB GPU with a small amount of CPU offload.

Qwen3.5-397B-A17B is currently the closest OSS model to frontier performance. The goal of this project is to make that model accessible for local deployment by reducing its memory footprint as much as possible without sacrificing the core capabilities that make it special.

---

5. Intended Use

Primary intended uses

• Research on:
  • MoE pruning and compression (especially REAP)
  • Scaling behavior of pruned MoE VL models
  • Trade-offs between expert sparsity and performance
• Experimental deployment for:
  • Vision–language assistants
  • Multimodal chatbots
  • Document + image understanding

Suitable tasks (examples)

• Multimodal chat (image + text → text)
• Image captioning / description
• Visual question answering
• General instruction-following and long-form text generation
• Reasoning and chain-of-thought tasks

Out-of-scope / high-risk uses

This model should not be used without additional safeguards for:

• Medical, legal, or financial advice
• Safety-critical decision making
• Political persuasion or targeted disinformation
• Any scenario where incorrect or biased outputs can cause real-world harm

---

6. Limitations & Risks

This model inherits all the limitations of Qwen3.5-397B-A17B plus those introduced by pruning:

• Hallucinations: The model can generate plausible but incorrect facts.
• Bias & toxicity: Biases from the original training data and synthetic calibration data remain and may be amplified.
• Distribution shift from pruning:
  • Some long-tail behaviors may degrade due to pruning 35% of experts — a more aggressive cut than previous REAP releases.
  • Performance may be uneven across tasks, domains, or languages not well covered in the calibration set.
• Multimodal edge cases:
  • Complex compositional visual reasoning or extremely high-resolution images may not work reliably.
  • VL behavior is preserved but not re-tuned after pruning.

Users should perform their own evaluation before relying on the model in any sensitive context.

---

7. How to Use

please check llama.cpp github

Recommended quantization: Q4_K_M or similar 4-bit quantization is recommended to fit within a single 96 GB GPU. A small amount of CPU offload may still be needed depending on your configuration.

---

8. Model Configuration Highlights

Key parameters after pruning:

• num_experts: 333 (down from 512)
• num_experts_per_tok: 10
• num_hidden_layers: 60
• hidden_size: 4096
• max_position_embeddings: 262,144
• Architecture: hybrid linear + full attention (full attention every 4 layers)
• Vision encoder: depth 27, hidden_size 1152, supports image and video tokens

---

9. Evaluation (Status)

• This release focuses on making the REAP-pruned model available for the community.
• Quantitative benchmarks (e.g., MMLU, reasoning, multimodal benchmarks) are still work in progress.
• Early qualitative checks show:
  • VL behavior is preserved after pruning.
  • Latency and memory usage are significantly improved compared to Qwen3.5-397B-A17B, enabling single-96GB-GPU deployment.

Community contributions with detailed benchmarks are very welcome.

---

10. Training & Distillation Details (High-Level)

• Base model: Qwen/Qwen3.5-397B-A17B
• Pruning method: REAP (Router-weighted Expert Activation Pruning)
• Expert count: 512 → 333 (~35% pruned)
• Calibration data: OpenMOSE/reap-calib-mix (mostly generated by Qwen3-235B-Instruct)
• Post-processing:
  • Router / gating structure retained
  • Experts pruned according to REAP scoring
  • No additional large-scale pretraining performed in this release

Future versions may include post-pruning fine-tuning or distillation to recover performance lost from the more aggressive pruning ratio.

---

11. Community & Contribution

Let's make frontier-class OSS models accessible together.

You are encouraged to:

• Run benchmarks and publish results
• Contribute scripts for:
  • Further pruning experiments
  • Quantization (e.g., GGUF, AWQ, GPTQ)
  • Long-context or domain-specific fine-tuning
• Report issues or findings about failure modes, biases, or surprising behaviors

---

12. License

• Model & code (this repository): Apache License 2.0
• The original Qwen3.5-397B-A17B model and any downstream use must also respect their respective licenses and usage terms.

---

13. Acknowledgements

This architecture research and implementation was made possible with computing power and technical support from Recursal AI. We sincerely thank them for enabling this work.

https://featherless.ai/

• Qwen team for building the Qwen3.5 family of models.
• Cerebras Research for the REAP method and reference implementation: https://github.com/CerebrasResearch/reap
• OpenMOSE community for experimentation, engineering, and calibration data generation.

---

2025 OpenMOSE