Spaces:

Soofi-Project
/

Pretraining-Tech-Report-old

Configuration error

App Files Files Community

mfromm commited on 9 days ago

Commit

d3585c9

verified ·

1 Parent(s): bc6374b

Upload 3 files

Browse files

Files changed (4) hide show

.gitattributes +1 -0
README.md +20 -6
index.html +240 -18
soofi_s_pretraining_report_v1.pdf +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+soofi_s_pretraining_report_v1.pdf filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,10 +1,24 @@
 ---
-title: Pretraining Tech Report
-emoji: 📈
-colorFrom: blue
-colorTo: green
 sdk: static
-pinned: false
 ---
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

 ---
+title: Soofi S — Sovereign German-English Foundation Model
+emoji: 🪶
+colorFrom: purple
+colorTo: blue
 sdk: static
+pinned: true
+license: apache-2.0
+short_description: Soofi S 30B-A3B pretraining report project page
 ---
+# Soofi S 30B-A3B — Project Page
+Static landing page for the Soofi S Pretraining Report v1.0.
+To deploy: create a new Space under https://huggingface.co/Soofi-Project with **SDK: Static**
+and upload `README.md`, `index.html`, and `soofi_s_pretraining_report_v1.pdf`
+(plus an optional `logo.png`). The "Paper (PDF)" button links to the PDF served
+directly from the Space.
+TODOs before publishing:
+- Once the preprint is on arXiv, swap the PDF button for an arXiv button and update the BibTeX entry.
+- Optionally swap the 🪶 emoji in the hero for the project wordmark (`logo.png`).

index.html CHANGED Viewed

@@ -1,19 +1,241 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
 </html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="description" content="Soofi S 30B-A3B: A sovereign, open-source foundation model for German and English.">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Soofi S: A Sovereign, Open-Source Foundation Model for German and English</title>
+  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css">
+  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
+  <style>
+    body { font-family: 'Noto Sans', sans-serif; }
+    /* soofipurple from the report: RGB(231,206,246) */
+    .hero.is-primary { background-color: #e7cef6; }
+    .hero.is-primary .title, .hero.is-primary .subtitle { color: #363636; }
+    .publication-authors { line-height: 1.6; }
+    .author-group { margin-top: 0.6em; }
+    .stat-box { border: 1px solid #e0e0e0; border-radius: 8px; padding: 1em; height: 100%; }
+    .stat-box .heading { color: #7a7a7a; }
+    .link-block { margin: 0.15em; display: inline-block; }
+    figure img { max-width: 100%; height: auto; }
+  </style>
+</head>
+<body>
+<section class="hero is-primary">
+  <div class="hero-body">
+    <div class="container has-text-centered">
+      <!-- Optional: replace the emoji with the project wordmark, e.g. <img src="logo.png" style="max-width:340px"> -->
+      <h1 class="title is-1">🪶 Soofi S 30B-A3B</h1>
+      <p class="subtitle is-5">A sovereign, open-source Mixture-of-Experts hybrid Mamba–Transformer foundation model for German and English</p>
+    </div>
+  </div>
+</section>
+<section class="hero">
+  <div class="hero-body">
+    <div class="container is-max-desktop">
+      <div class="columns is-centered">
+        <div class="column has-text-centered">
+          <h1 class="title is-2 publication-title">A Sovereign, Open-Source Foundation Model for German and English</h1>
+          <p class="subtitle is-5">Soofi S Pretraining Report v1.0</p>
+          <div class="is-size-5 publication-authors">
+            <p class="author-group"><strong>Core Team:</strong>
+              Benedikt Droste<sup>10</sup>, David Fitzek<sup>3,9</sup>, Ruben Härle<sup>5</sup>, Lukas Helff<sup>2,5</sup>,
+              Maximilian Idahl<sup>10</sup>, Alex Jude<sup>3,9</sup>, Abbas Goher Khan<sup>3</sup>, Maurice Kraus<sup>5</sup>,
+              Timm Ruland<sup>3,9</sup>, Richard Rutmann<sup>3,9</sup>, Sebastian Sztwiertnia<sup>5</sup>
+            </p>
+            <p class="author-group"><strong>Contributors:</strong>
+              Markus Frey<sup>3,9</sup>, Daniil Gurgurov<sup>2</sup>, Jan Pfister<sup>6</sup>, Tom Röhr<sup>7</sup>,
+              Sebastian von Rohrscheidt<sup>7</sup>
+            </p>
+            <p class="author-group"><strong>Advisors:</strong>
+              Jörg Bienert<sup>1</sup>, Nicolas Flores-Herr<sup>3</sup>, Simon Gottschalk<sup>8</sup>, Andreas Hotho<sup>6</sup>,
+              Kristian Kersting<sup>2,5,11</sup>, Joachim Köhler<sup>3</sup>, Alexander Löser<sup>7</sup>,
+              Wolfgang Nejdl<sup>8</sup>, Simon Ostermann<sup>2</sup>, Jan Plogsties<sup>4</sup>, Patrick Putzky<sup>12</sup>
+            </p>
+            <p class="author-group"><strong>Technical Leads:</strong>
+              Mehdi Ali<sup>3</sup>, Michael Fromm<sup>3</sup>, Max Lübbering<sup>3,9</sup>
+            </p>
+          </div>
+          <div class="is-size-6 publication-authors" style="margin-top: 1em;">
+            <span class="author-block"><sup>1</sup>KI Bundesverband,</span>
+            <span class="author-block"><sup>2</sup>DFKI,</span>
+            <span class="author-block"><sup>3</sup>Fraunhofer IAIS,</span>
+            <span class="author-block"><sup>4</sup>Fraunhofer IIS,</span>
+            <span class="author-block"><sup>5</sup>Technische Universität Darmstadt,</span>
+            <span class="author-block"><sup>6</sup>Universität Würzburg,</span>
+            <span class="author-block"><sup>7</sup>Berliner Hochschule für Technik,</span>
+            <span class="author-block"><sup>8</sup>L3S Research Center,</span>
+            <span class="author-block"><sup>9</sup>Lamarr,</span>
+            <span class="author-block"><sup>10</sup>ellamind,</span>
+            <span class="author-block"><sup>11</sup>hessian.AI,</span>
+            <span class="author-block"><sup>12</sup>Merantix Momentum</span>
+          </div>
+          <p class="is-size-6" style="margin-top: 1em;">
+            Consortium coordinated by the KI Bundesverband. Funded by the German Federal Ministry for
+            Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra.
+          </p>
+          <div class="column has-text-centered" style="margin-top: 0.5em;">
+            <!-- PDF hosted directly in this Space; swap for the arXiv button once the preprint is up -->
+            <span class="link-block">
+              <a href="soofi_s_pretraining_report_v1.pdf" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><i class="fas fa-file-pdf"></i></span>
+                <span>Paper (PDF)</span>
+              </a>
+            </span>
+            <span class="link-block">
+              <a href="https://huggingface.co/Soofi-Project" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><img src="https://huggingface.co/front/assets/huggingface_logo.svg" alt="Hugging Face" style="height: 1.0em; vertical-align: middle;"></span>
+                <span>Models &amp; Checkpoints</span>
+              </a>
+            </span>
+            <span class="link-block">
+              <a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><i class="fab fa-github"></i></span>
+                <span>Training &amp; Data Code</span>
+              </a>
+            </span>
+            <span class="link-block">
+              <a href="https://github.com/ellamind/base-eval" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><i class="fab fa-github"></i></span>
+                <span>Eval: base-eval</span>
+              </a>
+            </span>
+            <span class="link-block">
+              <a href="https://github.com/ellamind/eval-hive" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><i class="fab fa-github"></i></span>
+                <span>Eval: eval-hive</span>
+              </a>
+            </span>
+            <span class="link-block">
+              <a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank" class="external-link button is-normal is-rounded is-dark">
+                <span class="icon"><i class="fas fa-chart-line"></i></span>
+                <span>Training Logs (W&amp;B)</span>
+              </a>
+            </span>
+          </div>
+        </div>
+      </div>
+    </div>
+  </div>
+</section>
+<section class="section">
+  <div class="container content is-max-desktop">
+    <p>
+      <strong>Soofi S 30B-A3B</strong> is a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba–Transformer
+      foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and
+      keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense
+      models for long-context, high-concurrency deployment.
+    </p>
+    <p>
+      Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14–27B
+      models on aggregate English and German benchmarks, achieves the best code aggregates in both languages among
+      17 open base models, and outperforms every European sovereign baseline in our comparison — including ones far
+      larger in active parameters. Among fully open models, it obtains the highest English and German evaluation
+      scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud,
+      a sovereign HPC-scale AI infrastructure operated by Deutsche Telekom in Munich.
+    </p>
+  </div>
+</section>
+<section class="section pt-0">
+  <div class="container is-max-desktop">
+    <h2 class="title is-3">📌 At a Glance</h2>
+    <div class="columns is-multiline">
+      <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Parameters</p><p class="title is-4">31.6B total<br>3.2B active</p></div></div>
+      <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Architecture</p><p class="title is-4">52 layers<br>23 Mamba-2 · 23 MoE · 6 GQA</p></div></div>
+      <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Pretraining</p><p class="title is-4">~26.68T tokens<br>DE up to 15.3%</p></div></div>
+      <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Context</p><p class="title is-4">up to 1M<br>tokens</p></div></div>
+    </div>
+  </div>
+</section>
+<section class="section">
+  <div class="container content is-max-desktop">
+    <h2 class="title is-3">🧩 Highlights</h2>
+    <ol>
+      <li><strong>🏆 German–English champion:</strong> best English and German code aggregates among all 17 measured
+        open base models; strongest fully open model on the English and German aggregates; matches or outperforms
+        every European sovereign baseline on every German benchmark in our suite — at a fraction of the
+        active-parameter cost of dense 14–27B models.</li>
+      <li><strong>📋 Full data transparency:</strong> complete per-source, per-language token accounting for all
+        three training phases (including sources we evaluated and <em>excluded</em>), with reproducible corpus
+        construction scripts. ~99% of the mixture can be independently reconstructed.</li>
+      <li><strong>🔁 Reproducible recipe:</strong> full Warmup–Stable–Decay learning-rate schedule, optimizer, all
+        hyperparameters, per-phase token budgets, and phase boundaries — a third party can rebuild the run.</li>
+      <li><strong>⚡ Long-context serving efficiency:</strong> only 6 of 52 layers keep a KV cache, so the
+        per-sequence cache stays near-constant with context length. Measured aggregate decode TPS/GPU is 8–9× that
+        of dense 14–24B models at 40K context (batch 32) and stays essentially flat from 4K to 256K.</li>
+      <li><strong>🇩🇪 Sovereign end to end:</strong> trained from 24 March to 13 May 2026 on up to 512 NVIDIA B200
+        GPUs of the German Industrial AI Cloud (Deutsche Telekom, Munich), under European operational and
+        data-protection requirements.</li>
+    </ol>
+  </div>
+</section>
+<section class="section">
+  <div class="container content is-max-desktop">
+    <h2 class="title is-3">📊 Results</h2>
+    <ul>
+      <li><strong>Aggregates:</strong> English 70.1 / German 79.1 — highest among fully open models (Olmo 3 32B:
+        67.3 / 69.2; Apertus 70B: 62.4 / 72.8), on par with dense open-weight 14–27B models.</li>
+      <li><strong>Code:</strong> HumanEval 73.8, MBPP 70.2, MBPP-DE 84.2 — best in both comparison sets.</li>
+      <li><strong>Mathematics:</strong> GSM8K 86.1, GSM8K-Platinum-DE 87.1, Minerva-500 79.4.</li>
+      <li><strong>German:</strong> first on every German benchmark against European open-source baselines
+        (GLP-DE 88.8, INCLUDE-DE 61.2, ARC-Challenge-DE 92.3).</li>
+      <li><strong>Serving:</strong> 4.82k aggregate decode TPS/GPU at 40K context (batch 32, single B200, vLLM) —
+        9.2× Ministral 3 14B — with near-flat decode throughput from 4K to 256K.</li>
+    </ul>
+  </div>
+</section>
+<section class="section">
+  <div class="container content is-max-desktop">
+    <h2 class="title is-3">📁 Available Artifacts</h2>
+    <ul>
+      <li><a href="https://huggingface.co/Soofi-Project" target="_blank">🤗 Soofi S 30B-A3B Base weights and selected intermediate checkpoints</a></li>
+      <li><a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank">🛠️ Training code and reproducible data-construction scripts</a></li>
+      <li><a href="https://github.com/ellamind/base-eval" target="_blank">🧪 Evaluation code (base-eval)</a> and <a href="https://github.com/ellamind/eval-hive" target="_blank">eval-hive</a>, incl. parallel English/German suites</li>
+      <li><a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank">📈 Weights &amp; Biases dashboard of the full ~27T-token run</a></li>
+      <li>📋 Exact per-source token accounting for all three phases; commercially licensed sources (Genios) documented via aggregate statistics</li>
+    </ul>
+    <p>All artifacts are released under permissive licenses for transparent audit and extension.</p>
+  </div>
+</section>
+<section class="section">
+  <div class="container content is-max-desktop">
+    <h2 class="title is-3">📜 Citation</h2>
+    <p>If you use Soofi S or its artifacts, please cite the report
+      (<a href="soofi_s_pretraining_report_v1.pdf" target="_blank">PDF</a>):</p>
+    <!-- TODO: switch to the arXiv entry once the preprint is available -->
+    <pre><code>@techreport{soofi2026soofis,
+  title       = {A Sovereign, Open-Source Foundation Model for German and English: Soofi S Pretraining Report v1.0},
+  author      = {{The Soofi-Team}},
+  institution = {Project Soofi},
+  year        = {2026},
+  note        = {\url{https://huggingface.co/Soofi-Project}}
+}</code></pre>
+  </div>
+</section>
+<footer class="footer">
+  <div class="container content has-text-centered is-size-7">
+    <p>
+      Funded by the German Federal Ministry for Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra
+      through “Soofi: Souveräne KI für Europa” (grant no. 13IPC040A-J).
+      Website template adapted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a>/JQL project pages (Bulma).
+    </p>
+  </div>
+</footer>
+</body>
 </html>

soofi_s_pretraining_report_v1.pdf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:849ea26078b2a31f8132771d2ce8d179fced8c0c1986efb4e132086510a7aff6
+size 1601207