--- license: openmdw-1.1 license_link: https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md base_model: poolside/Laguna-S-2.1 base_model_relation: quantized library_name: mlx pipeline_tag: text-generation tags: - mlx - oq - quantized - moe - laguna --- # Laguna-S-2.1-oQ6e Calibrated 6-bit MLX quantization of [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) (118B total, 8B activated per token), produced with [oMLX](https://github.com/jundot/omlx) oQ at level 6 enhanced — **6.27 bits/weight effective**, 92 GB on disk. Data-driven mixed precision: bits are allocated per tensor from an imatrix-calibrated sensitivity map, not a fixed rule. For Apple Silicon. - **92 GB** on disk, down from 235 GB BF16 - 48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP); interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512) - Peak memory in my tests: **86.8 GB** at 1k context, 90.0 GB at 64k — you'll want a 128 GB Mac - Converted and tested on a **Macbook Pro M5 Max 128GB 40 GPU** ## Requirements mlx-lm doesn't support the `laguna` architecture yet — there's an open PR: [mlx-lm#1223](https://github.com/ml-explore/mlx-lm/pull/1223). Until it lands, use **mlx-vlm** (0.6.3+), which implements laguna as a text-only model: ```bash uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ6e --prompt "..." ``` oMLX serves it directly from **0.5.3** on — it vendors that PR and patches it into mlx-lm at import, so no model setting is needed. On earlier builds, discovery decides between the mlx-lm and mlx-vlm loaders by looking for a vision sub-config, and laguna has none — so it lands on mlx-lm and fails with `Model type laguna not supported`. Set **`model_type_override: "vlm"`** in the model's settings, then refresh discovery (`omlx restart`): the load failure is cached per entry until the next discovery pass, so setting the override alone won't clear it. ## Quantization oQ6e allocates bits per tensor from an importance-matrix calibration pass over calibration data. The 6-bit base lands on the experts; 248 of the 386 dense-spine tensors — attention, embeddings, `lm_head`, routers — were boosted to 8 bits. Output is standard MLX affine quantization — no custom kernels or runtime required. Like the oQ5e (and unlike the three smaller variants), this build was calibrated with omlx's newer adaptive imatrix collection: up to 1024 samples, extended until every routed expert is covered, instead of a fixed 128 samples. ## How it was quantized oQ at level 6 enhanced — imatrix-calibrated, group size 128. I had to patch omlx to route laguna through the mlx-vlm loader. At 235 GB the model doesn't fit in 128 GB of RAM, so calibration ran against a uniform 4-bit proxy on disk rather than the FP weights, which shifts the bit allocation slightly. ## Conversion check Smoke-tested after conversion with `mlx_vlm.generate`: coherent — solved `17 * 24 = 408`, no repetition loop. 42.4 tok/s generation on a short prompt, peak 92.6 GB. ## Performance Measured with oMLX's benchmark harness on a **Macbook Pro M5 Max 128GB 40 GPU**, single request, 128 generated tokens: | prompt | gen tok/s | prefill tok/s | TTFT ms | peak GB | |---|---|---|---|---| | 1k | 53.0 | 940.3 | 1090 | 86.83 | | 4k | 53.1 | 996.0 | 4114 | 86.98 | | 8k | 50.1 | 876.9 | 9343 | 87.17 | | 16k | 47.4 | 799.2 | 20502 | 87.54 | | 32k | 43.1 | 731.3 | 44807 | 88.33 | | 64k | 32.9 | 575.5 | 113871 | 89.95 | Continuous batching at 1k prompt / 128 generated: | batch | tg tok/s | speedup | TTFT ms | E2E s | |---|---|---|---|---| | 1 | 53.0 | 1.00x | 1090 | 3.51 | | 2 | 71.0 | 1.34x | 2210 | 5.81 | | 4 | 90.1 | 1.70x | 4275 | 10.06 | | 8 | 114.0 | 2.15x | 5667 | 16.20 | ## Benchmarks & Variants mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off, identical questions across every variant. The bf16 row is the hosted API, measured the same way. Standard error at this n is around 2.5 points, so oQ4e through oQ6e aren't separated by this run. ![Accuracy vs bits per weight, three benchmarks, n=300](ladder.png) | Variant | Size | bpw | gen tok/s (1k → 64k) | mmlu_pro | mathqa | winogrande | |---|---|---|---|---|---|---| | [Laguna-S-2.1-oQ2e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e-fast) | 35 GB | 2.60 | 78.8 → 48.6 | 0.700 | 0.850 | 0.713 | | [Laguna-S-2.1-oQ2e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ2e) | 36 GB | 2.70 | 61.5 → 38.8 | 0.703 | 0.840 | 0.707 | | [Laguna-S-2.1-oQ3e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e-fast) | 49 GB | 3.56 | 77.2 → 48.4 | 0.750 | 0.887 | 0.760 | | [Laguna-S-2.1-oQ3e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ3e) | 49 GB | 3.59 | 67.5 → 40.1 | 0.750 | 0.880 | 0.760 | | [Laguna-S-2.1-oQ4e-fast](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e-fast) | 63 GB | 4.54 | 69.3 → 45.5 | 0.787 | 0.873 | 0.777 | | [Laguna-S-2.1-oQ4e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e) | 64 GB | 4.60 | 55.9 → 39.8 | 0.757 | 0.887 | 0.777 | | [Laguna-S-2.1-oQ5e](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ5e) | 78 GB | 5.30 | 57.5 → 38.1 | 0.773 | 0.883 | 0.797 | | [**Laguna-S-2.1-oQ6e**](https://huggingface.co/mlx-community/Laguna-S-2.1-oQ6e) (this repo) | 92 GB | 6.27 | 53.0 → 32.9 | 0.763 | 0.873 | 0.777 | | [Laguna S 2.1 (API, bf16)](https://openrouter.ai/poolside/laguna-s-2.1) | — | 16 | — | 0.773 | 0.880 | 0.810 | Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what the model does — no long-context work, no agentic loops, no real code — and at this sample size most of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels, not to rank the variants against each other. Test the one you're considering on your own workload before trusting any of it. ## Usage ```bash # mlx-vlm — plain mlx-lm doesn't support the laguna architecture uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ6e \ --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300 # oMLX — discovers the model from the HF cache; set model_type_override: "vlm" first omlx serve ``` ## License [OpenMDW-1.1](https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md), inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.