Spaces:
Configuration error
Configuration error
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="description" content="Soofi S 30B-A3B: A sovereign, open-source foundation model for German and English."> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Soofi S: A Sovereign, Open-Source Foundation Model for German and English</title> | |
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> | |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css"> | |
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css"> | |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> | |
| <style> | |
| body { font-family: 'Noto Sans', sans-serif; } | |
| /* soofipurple from the report: RGB(231,206,246) */ | |
| .hero.is-primary { background-color: #e7cef6; } | |
| .hero.is-primary .title, .hero.is-primary .subtitle { color: #363636; } | |
| .publication-authors { line-height: 1.6; } | |
| .author-group { margin-top: 0.6em; } | |
| .stat-box { border: 1px solid #e0e0e0; border-radius: 8px; padding: 1em; height: 100%; } | |
| .stat-box .heading { color: #7a7a7a; } | |
| .link-block { margin: 0.15em; display: inline-block; } | |
| figure img { max-width: 100%; height: auto; } | |
| </style> | |
| </head> | |
| <body> | |
| <section class="hero is-primary"> | |
| <div class="hero-body"> | |
| <div class="container has-text-centered"> | |
| <!-- Optional: replace the emoji with the project wordmark, e.g. <img src="logo.png" style="max-width:340px"> --> | |
| <img src="logo.png" alt="Soofi S" style="max-width: 340px; height: auto;"> | |
| <p class="subtitle is-5">A sovereign, open-source Mixture-of-Experts hybrid Mamba–Transformer foundation model for German and English</p> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="hero"> | |
| <div class="hero-body"> | |
| <div class="container is-max-desktop"> | |
| <div class="columns is-centered"> | |
| <div class="column has-text-centered"> | |
| <h1 class="title is-2 publication-title">A Sovereign, Open-Source Foundation Model for German and English</h1> | |
| <p class="subtitle is-5">Soofi S Pretraining Report v1.0</p> | |
| <div class="is-size-4 publication-authors"> | |
| <p class="author-group"><strong>The Soofi-Team</strong></p> | |
| </div> | |
| <div class="is-size-6 publication-authors" style="margin-top: 1em;"> | |
| <span class="author-block">KI Bundesverband,</span> | |
| <span class="author-block">DFKI,</span> | |
| <span class="author-block">Fraunhofer IAIS,</span> | |
| <span class="author-block">Fraunhofer IIS,</span> | |
| <span class="author-block">Technische Universität Darmstadt,</span> | |
| <span class="author-block">Universität Würzburg,</span> | |
| <span class="author-block">Berliner Hochschule für Technik,</span> | |
| <span class="author-block">L3S Research Center,</span> | |
| <span class="author-block">Lamarr,</span> | |
| <span class="author-block">ellamind,</span> | |
| <span class="author-block">hessian.AI,</span> | |
| <span class="author-block">Merantix Momentum</span> | |
| </div> | |
| <p class="is-size-6" style="margin-top: 1em;"> | |
| Consortium coordinated by the KI Bundesverband. Funded by the German Federal Ministry for | |
| Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra. | |
| </p> | |
| <div class="column has-text-centered" style="margin-top: 0.5em;"> | |
| <!-- PDF hosted directly in this Space; swap for the arXiv button once the preprint is up --> | |
| <span class="link-block"> | |
| <a href="soofi_s_pretraining_report_v1.pdf" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><i class="fas fa-file-pdf"></i></span> | |
| <span>Paper (PDF)</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://huggingface.co/Soofi-Project" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><img src="https://huggingface.co/front/assets/huggingface_logo.svg" alt="Hugging Face" style="height: 1.0em; vertical-align: middle;"></span> | |
| <span>Models & Checkpoints</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><i class="fab fa-github"></i></span> | |
| <span>Training & Data Code</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://github.com/ellamind/base-eval" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><i class="fab fa-github"></i></span> | |
| <span>Eval: base-eval</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://github.com/ellamind/eval-hive" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><i class="fab fa-github"></i></span> | |
| <span>Eval: eval-hive</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank" class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"><i class="fas fa-chart-line"></i></span> | |
| <span>Training Logs (W&B)</span> | |
| </a> | |
| </span> | |
| </div> | |
| <div class="notification is-warning is-light" style="margin-top: 1em;"> | |
| <strong>Note:</strong> During the current beta phase, the model repositories are gated — | |
| you need to accept the access conditions on Hugging Face before downloading. | |
| Once the beta phase ends, the models will be freely available without access request. | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="section"> | |
| <div class="container content is-max-desktop"> | |
| <p> | |
| <strong>Soofi S 30B-A3B</strong> is a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba–Transformer | |
| foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and | |
| keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense | |
| models for long-context, high-concurrency deployment. | |
| </p> | |
| <p> | |
| Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14–27B | |
| models on aggregate English and German benchmarks, achieves the best code aggregates in both languages among | |
| 17 open base models, and outperforms every European sovereign baseline in our comparison — including ones far | |
| larger in active parameters. Among fully open models, it obtains the highest English and German evaluation | |
| scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, | |
| a sovereign HPC-scale AI infrastructure operated by Deutsche Telekom in Munich. | |
| </p> | |
| </div> | |
| </section> | |
| <section class="section pt-0"> | |
| <div class="container is-max-desktop"> | |
| <h2 class="title is-3">📌 At a Glance</h2> | |
| <div class="columns is-multiline"> | |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Parameters</p><p class="title is-4">31.6B total<br>3.2B active</p></div></div> | |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Architecture</p><p class="title is-4">52 layers<br>23 Mamba-2 · 23 MoE · 6 GQA</p></div></div> | |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Pretraining</p><p class="title is-4">~26.68T tokens<br>DE up to 15.3%</p></div></div> | |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Context</p><p class="title is-4">up to 1M<br>tokens</p></div></div> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="section"> | |
| <div class="container content is-max-desktop"> | |
| <h2 class="title is-3">🧩 Highlights</h2> | |
| <ol> | |
| <li><strong>🏆 German–English champion:</strong> best English and German code aggregates among all 17 measured | |
| open base models; strongest fully open model on the English and German aggregates; matches or outperforms | |
| every European sovereign baseline on every German benchmark in our suite — at a fraction of the | |
| active-parameter cost of dense 14–27B models.</li> | |
| <li><strong>📋 Full data transparency:</strong> complete per-source, per-language token accounting for all | |
| three training phases (including sources we evaluated and <em>excluded</em>), with reproducible corpus | |
| construction scripts. ~99% of the mixture can be independently reconstructed.</li> | |
| <li><strong>🔁 Reproducible recipe:</strong> full Warmup–Stable–Decay learning-rate schedule, optimizer, all | |
| hyperparameters, per-phase token budgets, and phase boundaries — a third party can rebuild the run.</li> | |
| <li><strong>⚡ Long-context serving efficiency:</strong> only 6 of 52 layers keep a KV cache, so the | |
| per-sequence cache stays near-constant with context length. Measured aggregate decode TPS/GPU is 8–9× that | |
| of dense 14–24B models at 40K context (batch 32) and stays essentially flat from 4K to 256K.</li> | |
| <li><strong>🇩🇪 Sovereign end to end:</strong> trained from 24 March to 13 May 2026 on up to 512 NVIDIA B200 | |
| GPUs of the German Industrial AI Cloud (Deutsche Telekom, Munich), under European operational and | |
| data-protection requirements.</li> | |
| </ol> | |
| </div> | |
| </section> | |
| <section class="section"> | |
| <div class="container content is-max-desktop"> | |
| <h2 class="title is-3">📊 Results</h2> | |
| <ul> | |
| <li><strong>Aggregates:</strong> English 70.1 / German 79.1 — highest among fully open models (Olmo 3 32B: | |
| 67.3 / 69.2; Apertus 70B: 62.4 / 72.8), on par with dense open-weight 14–27B models.</li> | |
| <li><strong>Code:</strong> HumanEval 73.8, MBPP 70.2, MBPP-DE 84.2 — best in both comparison sets.</li> | |
| <li><strong>Mathematics:</strong> GSM8K 86.1, GSM8K-Platinum-DE 87.1, Minerva-500 79.4.</li> | |
| <li><strong>German:</strong> first on every German benchmark against European open-source baselines | |
| (GLP-DE 88.8, INCLUDE-DE 61.2, ARC-Challenge-DE 92.3).</li> | |
| <li><strong>Serving:</strong> 4.82k aggregate decode TPS/GPU at 40K context (batch 32, single B200, vLLM) — | |
| 9.2× Ministral 3 14B — with near-flat decode throughput from 4K to 256K.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <section class="section"> | |
| <div class="container content is-max-desktop"> | |
| <h2 class="title is-3">📁 Available Artifacts</h2> | |
| <ul> | |
| <li><a href="https://huggingface.co/Soofi-Project" target="_blank">🤗 Soofi S 30B-A3B Base weights and selected intermediate checkpoints</a> | |
| <em>(currently gated during the beta phase; freely available once the beta ends)</em></li> | |
| <li><a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank">🛠️ Training code and reproducible data-construction scripts</a></li> | |
| <li><a href="https://github.com/ellamind/base-eval" target="_blank">🧪 Evaluation code (base-eval)</a> and <a href="https://github.com/ellamind/eval-hive" target="_blank">eval-hive</a>, incl. parallel English/German suites</li> | |
| <li><a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank">📈 Weights & Biases dashboard of the full ~27T-token run</a></li> | |
| <li>📋 Exact per-source token accounting for all three phases; commercially licensed sources (Genios) documented via aggregate statistics</li> | |
| </ul> | |
| <p>All artifacts are released under permissive licenses for transparent audit and extension.</p> | |
| </div> | |
| </section> | |
| <section class="section"> | |
| <div class="container content is-max-desktop"> | |
| <h2 class="title is-3">📜 Citation</h2> | |
| <p>If you use Soofi S or its artifacts, please cite the report | |
| (<a href="soofi_s_pretraining_report_v1.pdf" target="_blank">PDF</a>):</p> | |
| <!-- TODO: switch to the arXiv entry once the preprint is available --> | |
| <pre><code>@techreport{soofi2026soofis, | |
| title = {A Sovereign, Open-Source Foundation Model for German and English: Soofi S Pretraining Report v1.0}, | |
| author = {{The Soofi-Team}}, | |
| institution = {Project Soofi}, | |
| year = {2026}, | |
| note = {\url{https://huggingface.co/Soofi-Project}} | |
| }</code></pre> | |
| </div> | |
| </section> | |
| <footer class="footer"> | |
| <div class="container content has-text-centered is-size-7"> | |
| <p> | |
| Funded by the German Federal Ministry for Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra | |
| through “Soofi: Souveräne KI für Europa” (grant no. 13IPC040A-J). | |
| Website template adapted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a>/JQL project pages (Bulma). | |
| </p> | |
| </div> | |
| </footer> | |
| </body> | |
| </html> |