Text Generation
MLX
GGUF
1b
1b-active
5b
7b
allenai
android
apple-silicon
attested
calibration-aware-pruning
chain-of-custody
chinese
consumer-gpu
cryptographically-verified
edge-inference
embedded
english
expert-pruning
forge-alloy
fully-open
general
general-purpose
ggml
iphone
llama-cpp
lm-studio
local-inference
macbook
mixture-of-experts
mobile
Mixture of Experts
multilingual
ollama
olmoe
on-device
q5-k-m
q5_k_m
quantized
raspberry-pi
reproducible
sparse-moe
versatile
conversational
Instructions to use continuum-ai/olmoe-1b-7b-compacted-5b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use continuum-ai/olmoe-1b-7b-compacted-5b with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("continuum-ai/olmoe-1b-7b-compacted-5b") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use continuum-ai/olmoe-1b-7b-compacted-5b with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M # Run inference directly in the terminal: llama cli -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M # Run inference directly in the terminal: llama cli -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M # Run inference directly in the terminal: ./llama-cli -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
Use Docker
docker model run hf.co/continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
- LM Studio
- Jan
- vLLM
How to use continuum-ai/olmoe-1b-7b-compacted-5b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "continuum-ai/olmoe-1b-7b-compacted-5b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "continuum-ai/olmoe-1b-7b-compacted-5b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
- Ollama
How to use continuum-ai/olmoe-1b-7b-compacted-5b with Ollama:
ollama run hf.co/continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
- Unsloth Studio
How to use continuum-ai/olmoe-1b-7b-compacted-5b with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for continuum-ai/olmoe-1b-7b-compacted-5b to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for continuum-ai/olmoe-1b-7b-compacted-5b to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for continuum-ai/olmoe-1b-7b-compacted-5b to start chatting
- Atomic Chat new
- MLX LM
How to use continuum-ai/olmoe-1b-7b-compacted-5b with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "continuum-ai/olmoe-1b-7b-compacted-5b"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "continuum-ai/olmoe-1b-7b-compacted-5b" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "continuum-ai/olmoe-1b-7b-compacted-5b", "messages": [ {"role": "user", "content": "Hello"} ] }' - Docker Model Runner
How to use continuum-ai/olmoe-1b-7b-compacted-5b with Docker Model Runner:
docker model run hf.co/continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
- Lemonade
How to use continuum-ai/olmoe-1b-7b-compacted-5b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull continuum-ai/olmoe-1b-7b-compacted-5b:Q5_K_M
Run and chat with the model
lemonade run user.olmoe-1b-7b-compacted-5b-Q5_K_M
List all available models
lemonade list
card: add org front-door footer v3
Browse files
README.md
CHANGED
|
@@ -148,6 +148,68 @@ The Factory configurator lets you design and forge custom models visually β co
|
|
| 148 |
|
| 149 |
[GitHub](https://github.com/CambrianTech/continuum) Β· [All Models](https://huggingface.co/continuum-ai) Β· [Forge-Alloy](https://github.com/CambrianTech/forge-alloy)
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
## License
|
| 152 |
|
| 153 |
apache-2.0
|
|
|
|
| 148 |
|
| 149 |
[GitHub](https://github.com/CambrianTech/continuum) Β· [All Models](https://huggingface.co/continuum-ai) Β· [Forge-Alloy](https://github.com/CambrianTech/forge-alloy)
|
| 150 |
|
| 151 |
+
---
|
| 152 |
+
|
| 153 |
+
## More from continuum-ai
|
| 154 |
+
|
| 155 |
+
`continuum-ai` ships **structurally compacted models for hardware tiers nobody else targets**. Every artifact is calibration-aware, hardware-anchored, and shipped with [ForgeAlloy](https://github.com/CambrianTech/forge-alloy) cryptographic provenance β the per-problem benchmark JSONLs are uploaded with sha256 hashes recorded in the alloy so anyone can re-score against the same anchor without trusting the producer's claim.
|
| 156 |
+
|
| 157 |
+
### Currently shipped
|
| 158 |
+
|
| 159 |
+
| Model | Base | HumanEval (vs base) | Tier | What's new |
|
| 160 |
+
|---|---|---|---|---|
|
| 161 |
+
| [**qwen3-coder-30b-a3b-compacted-19b-256k**](https://huggingface.co/continuum-ai/qwen3-coder-30b-a3b-compacted-19b-256k) | Qwen3-Coder-30B-A3B-Instruct | **88.4** (base 92.1, Ξ β3.7) | **12 GB Q4_K_M** | First 30B-class coder that fits a 12 GB consumer GPU. Calibration-aware MoE expert pruning (Β§4.1.3.4). 256K context. |
|
| 162 |
+
| [**qwen2.5-coder-7b-compacted**](https://huggingface.co/continuum-ai/qwen2.5-coder-7b-compacted) | Qwen2.5-Coder-7B | 61.0 (base 62.2, Ξ β1.2) | 16 GB fp16 | Methodology validation artifact for Β§4.1.3.3 β compensation LoRA closes the dense-head pruning gap to within Β±3pt of base. |
|
| 163 |
+
| [**olmoe-1b-7b-compacted-5b**](https://huggingface.co/continuum-ai/olmoe-1b-7b-compacted-5b) | OLMoE-1B-7B-0924-Instruct (Allen AI, fully open) | **36.0** (base 40.9, Ξ β4.9) | **4 GB Q5_K_M / phone tier** | Cross-architecture validation of Β§4.1.3.4 β same forge scripts ported `Qwen3MoeForCausalLM` β `OlmoeForCausalLM` without modification. The +8.0 within-model swing between broad-corpus and code-corpus calibration is the second empirical anchor for the discipline gate. |
|
| 164 |
+
|
| 165 |
+
### Forge methodology in one paragraph
|
| 166 |
+
|
| 167 |
+
A prunable unit's importance MUST be derived from **task-conditioned activation profiling on a held-out corpus** that reflects the artifact's intended workload. Architectural-only metrics (router gate norms, weight norms, magnitudes) are first-pass shortcuts that systematically underperform task-specific activation metrics β empirically validated at two structurally distinct units (dense heads in Β§4.1.3.1, MoE experts in Β§4.1.3.4) with a +9.7 HumanEval swing on the same prune budget. **Get the metric right AND the calibration corpus right; the artifact follows.** Two discipline gates now derived from empirical failures, not asserted from first principles: **Β§4.1.4.1 anchor-reproduction gate** (the base anchor must reproduce within Β±3pt on the publishing pipeline before any calibrated delta is reported), and **Β§4.1.3.4.1 calibration-corpus discipline gate** (the calibration corpus used for importance profiling must be hash-pinned in the alloy AND must be a representative sample of the eval workload distribution β wrong-corpus and wrong-metric saturate at the same ~13 HumanEval damage ceiling, demonstrated empirically across two architectures). Full methodology in [PLASTICITY-COMPACTION.md](https://github.com/CambrianTech/continuum/blob/main/docs/papers/PLASTICITY-COMPACTION.md).
|
| 168 |
+
|
| 169 |
+
### The empty-quadrant frontier
|
| 170 |
+
|
| 171 |
+
A live HuggingFace audit (April 2026) confirmed that **the entire structurally-pruned-MoE quadrant is empty for every frontier model except Llama 3.3 70B**. Quantization is everywhere; structural pruning is nowhere. The forge methodology validated on `qwen3-coder-30b-a3b` ports directly to every other MoE family. The forge queue below is the comprehensive map of empty quadrants we are claiming, one architecture at a time.
|
| 172 |
+
|
| 173 |
+
### Forge queue β comprehensive new-architecture coverage
|
| 174 |
+
|
| 175 |
+
| # | Target | Arch | License | Total/Active | Tier post-prune | Status |
|
| 176 |
+
|---|---|---|---|---|---|---|
|
| 177 |
+
| 1 | OLMoE-1B-7B (`OlmoeForCausalLM`) | `OlmoeForCausalLM` | Apache-2.0 | 7B/1.3B β 5B/1.0B | **Phone / 4 GB Q5** | β
**SHIPPED** as `olmoe-1b-7b-compacted-5b`. Second cross-arch validation of Β§4.1.3.4. |
|
| 178 |
+
| 2 | [ibm-granite/granite-3.1-3b-a800m-instruct](https://huggingface.co/ibm-granite/granite-3.1-3b-a800m-instruct) | `GraniteMoeForCausalLM` | Apache-2.0 | 3.3B/800M (40e/top-8) | Edge tier | **Downloading now.** IBM enterprise brand, ultra-rare tiny-MoE niche, zero pruned variants. |
|
| 179 |
+
| 3 | [deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat) | `DeepseekV2ForCausalLM` | DeepSeek (commercial OK) | 15.7B/2.4B | Single GPU | **Downloading now.** The forgotten DeepSeek sibling β DeepSeek brand without 670 GB of VRAM. |
|
| 180 |
+
| 4 | [microsoft/Phi-3.5-MoE-instruct](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct) | `PhiMoEForCausalLM` | **MIT** | 42B/6.6B (16e/top-2) | Single 5090 Q4 | Queued. MIT-licensed Microsoft MoE that nobody runs because 42B is the awkward middle tier β until you prune to 12 experts. |
|
| 181 |
+
| 5 | [mistralai/Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) | `MixtralForCausalLM` | Apache-2.0 | 141B/39B (8e/top-2) | Single 5090 Q4 | Queued. Two-year overdue Pareto win β the textbook MoE that nobody has ever calibration-pruned. |
|
| 182 |
+
| 6 | [Qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | `Qwen3MoeForCausalLM` | Apache-2.0 | 235B/22B (128e/top-8) | Single 5090 Q4 | Queued. Same family as our shipped 30B-A3B β methodology ports trivially. |
|
| 183 |
+
| 7 | [Qwen/Qwen3-Coder-480B-A35B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct) | `Qwen3MoeForCausalLM` | Apache-2.0 | 480B/35B (160e/top-8) | **Grid moonshot** (4Γ24GB) | Queued. First consumer-accessible 480B coder. |
|
| 184 |
+
| 8 | [deepseek-ai/DeepSeek-Coder-V2-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) | `DeepseekV2ForCausalLM` | DeepSeek | 236B/21B | Grid | Queued. Direct methodology replay at higher tier. |
|
| 185 |
+
| 9 | [Snowflake/snowflake-arctic-instruct](https://huggingface.co/Snowflake/snowflake-arctic-instruct) | `ArcticForCausalLM` | Apache-2.0 | 480B/17B (128e/top-2) | Grid | Queued. The forgotten Apache frontier MoE β dense+sparse hybrid arch is a novel research contribution by itself. |
|
| 186 |
+
| 10 | [deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) | `DeepseekV3ForCausalLM` | **MIT** | 671B/37B | **Grid moonshot** | Queued. The viral king. First non-distill R1 compaction. |
|
| 187 |
+
|
| 188 |
+
**8 distinct architecture classes** covered across **5 hardware tiers** (edge β phone β single GPU β 5090 β grid). When the queue completes, the calibration-aware-importance metric has been validated on `Qwen3MoeForCausalLM`, `OlmoeForCausalLM`, `GraniteMoeForCausalLM`, `DeepseekV2ForCausalLM`, `PhiMoEForCausalLM`, `MixtralForCausalLM`, `ArcticForCausalLM`, and `DeepseekV3ForCausalLM` β the cross-family invariance claim becomes empirical, not theoretical.
|
| 189 |
+
|
| 190 |
+
### Hard prerequisites being built in parallel
|
| 191 |
+
|
| 192 |
+
- **LiveCodeBench v6 anchor extension** for `eval_with_calibration.py` β HumanEval is no longer reported on frontier model cards (Qwen3-Coder, DeepSeek-V3.1, Mixtral 8x22B all use SWE-bench / LiveCodeBench / Aider-Polyglot). Without LCB v6 wired up, frontier targets are blocked at the Β§4.1.4.1 calibration discipline gate. ~1-2 days of mechanical pipeline work.
|
| 193 |
+
- **Offline teacher-logit precomputation** for `compensation_lora.py` β at 30B+ class, transformers' `caching_allocator_warmup` pre-allocates an fp16 buffer equal to full model size before bnb 4-bit takes effect, exceeding total VRAM on a single 32 GB GPU. The architecturally correct fix is phase-1-load-teacher / phase-2-unload / phase-3-load-student-and-train-against-on-disk-logits. Prerequisite for compensation v2 of every artifact β₯30B.
|
| 194 |
+
- **Grid expert sharding** for the 480B+ moonshots β `cpu_expert_prune_v2.py`'s streaming pruner already handles shards bigger than any single GPU, but distributed inference + cross-machine activation profiling for the calibration-aware metric needs the grid layer. This is the Β§4.1.3.5 distributed forge methodology paper section.
|
| 195 |
+
|
| 196 |
+
### Sensory bridge stack (separate from the LLM forge queue)
|
| 197 |
+
|
| 198 |
+
For Continuum's own sensory architecture (vision/audio/embedding bridges), the right targets are not forge candidates β they're curated bridge components used as-is:
|
| 199 |
+
|
| 200 |
+
| Component | Model | Use |
|
| 201 |
+
|---|---|---|
|
| 202 |
+
| Vision encoder | [`google/siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384) | Image embeddings for the vision bridge |
|
| 203 |
+
| Vision describer | [`microsoft/Phi-3.5-vision-instruct`](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) | Small VLM that generates text descriptions consumed by text-only LLMs |
|
| 204 |
+
| STT | [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3) | Speech transcription for audio bridge |
|
| 205 |
+
| Multilingual embedding | [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) | Sensory cache embeddings |
|
| 206 |
+
| Avatar diffusion | [`black-forest-labs/FLUX.1-schnell`](https://huggingface.co/black-forest-labs/FLUX.1-schnell) | Apache-licensed avatar generation for Continuum universes |
|
| 207 |
+
|
| 208 |
+
### What we DON'T target
|
| 209 |
+
|
| 210 |
+
The Llama 3.3 70B slot is saturated (six publishers, every quant level). We're not shipping a third compacted MoE in the middle tier. The lab's brand pitch is **models that no individual hardware tier can run, made runnable by structural compaction + grid distribution** β empty-quadrant headlines, not catalog filler. That's the intersection only continuum has, and the forge queue above is the map.
|
| 211 |
+
|
| 212 |
+
|
| 213 |
## License
|
| 214 |
|
| 215 |
apache-2.0
|