Update standalone Apertus embedding model: moe_head

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,5 @@
+{
+    "embedding_dimension": 1024,
+    "pooling_mode": "mean",
+    "include_prompt": true
+}

2_LanguageMoE/config.json

@@ -0,0 +1,15 @@
+{
+    "hidden_size": 1024,
+    "embedding_dim": 1024,
+    "expert_names": [
+        "de",
+        "en",
+        "fr",
+        "it",
+        "rm",
+        "gsw",
+        "shared"
+    ],
+    "top_k": 2,
+    "temperature": 1.0
+}

language_moe_head.pt → 2_LanguageMoE/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:58d2b1243e87e4d86d8d90606b08fd486689bc096f86118beccc874d6fb93952
-size 29392735
+oid sha256:c7544c30354147275e494c847a59ff011e09ef467210ff64c6bfd7d11b941caf
+size 14695206

README.md

@@ -1,5 +1,8 @@
 ---
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
 license: apache-2.0
+base_model: andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir
 datasets:
 - wikimedia/wikipedia
 - eljuanina/VotingBooklets-v1
@@ -12,52 +15,177 @@ language:
 - rm
 - gsw
 tags:
+- sentence-transformers
 - embeddings
 - retrieval
 - multilingual
 - swiss
 - apertus
+- apertus-1.1
 - bidirectional
 - matryoshka
+- moe
+- language-moe
+- sparse-routing
 ---
 
-# Apertus Swiss Embedding Experiment — moe_head
+# Apertus v1.1 Swiss Embed — Language-MoE
+
+A standalone Sentence Transformers retrieval model based on
+[`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir).
+
+The model retains the trained **dense bidirectional Apertus encoder backbone** and replaces
+the parent's single dense embedding projection with a learned **Language-MoE projection
+head**.
+
+> This release is a **dense-backbone + sparse Language-MoE head**, not a backbone-level
+> native MoE model.
+
+## Architecture
+
+```text
+plain retrieval text
+        ↓
+bidirectional Apertus encoder
+        ↓
+mean pooling
+        ↓
+Language-MoE router
+        ↓
+Top-2 of:
+DE / EN / FR / IT / RM / GSW / shared
+        ↓
+1024d projection
+        ↓
+L2 normalization
+```
+
+| Property | Value |
+|---|---|
+| Total parameters | **445,696,559 (0.446B)** |
+| Active parameters / sentence | **440,453,679** |
+| Language-MoE head parameters | **7,347,207** |
+| Active head parameters / sentence | **2,104,327** |
+| Router parameters | 7,175 |
+| Parameters / expert | 1,048,576 |
+| Experts | 7 |
+| Active experts / sentence | 2 |
+| Native embedding dimension | **1024** |
+| Matryoshka dimensions | `[1024, 768, 512, 256]` |
+| MoE training max length | **256 tokens** |
+| Inference max length | **1024 tokens** |
+| Sparse/reference max abs diff | `1.907e-06` |
+
+## Sentence Transformers usage
+
+`LanguageMoE` is a custom Sentence Transformers module, so load the trusted repository with
+`trust_remote_code=True`:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer(
+    "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
+    trust_remote_code=True,
+)
+
+queries = model.encode_query([
+    "Welche Aufgaben hat der Bundesrat?"
+])
+
+documents = model.encode_document([
+    "Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
+    "Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
+])
+
+scores = model.similarity(queries, documents)
+print(scores)
+```
+
+For a compact Matryoshka representation:
+
+```python
+embeddings = model.encode_query(
+    texts,
+    truncate_dim=512,
+)
+```
+
+The stored retrieval prefixes are:
 
-Language-MoE projection with 7 experts (de, en, fr, it, rm, gsw, shared), learned Top-2 routing. Base model: `andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir` at revision `a6f4b025ee825eec8be04d1132c9e24f20841778`.
+- query: `query: `
+- document: `passage: `
 
-Effective parameters: **445,696,559**
+No chat template is used.
 
-Training profile: **`size_optimized`**
+## Lineage
 
-## Internal diagnostics
+| Property | Value |
+|---|---|
+| Original source | [`swiss-ai/Apertus-v1.1-0.5B`](https://huggingface.co/swiss-ai/Apertus-v1.1-0.5B) |
+| Dense embedding parent | [`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir) |
+| Dense parent revision | `a6f4b025ee825eec8be04d1132c9e24f20841778` |
+| Apertus lineage | [`swiss-ai/Apertus-8B-2509`](https://huggingface.co/swiss-ai/Apertus-8B-2509) |
 
-| Set | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
-|---|---:|---:|---:|---:|---:|
-| Swiss mono | 1024 | 0.8125 | 0.9229 | 0.8665 | 0.8487 |
-| Swiss mono | 512 | 0.8063 | 0.9208 | 0.8639 | 0.8458 |
-| Swiss cross | 1024 | 0.3083 | 0.9667 | 0.6669 | 0.5672 |
-| Swiss cross | 512 | 0.3104 | 0.9667 | 0.6670 | 0.5675 |
+The repository contains the complete transformer/tokenizer/pooling/MoE/normalization pipeline.
+The dense parent is **not fetched at inference time**.
 
-These are internal development diagnostics, not MTEB/MMTEB claims.
+## Language-MoE training
 
-## Data
+The dense embedding backbone was frozen. Its pooled hidden states were cached once and the
+router + expert projections were trained for the controlled MoE-head ablation.
 
-Training: `wikimedia/wikipedia`, `eljuanina/VotingBooklets-v1`, `ZurichNLP/SwissGov-RSD`.
+See `training_metadata.json` for the exact cache/training history and
+`router_diagnostics.json` for held-out routing counts.
 
-`eljuanina/VotingBooklets-Diamond-v1` is evaluation-only.
+Training data:
 
-## Retrieval preprocessing
+- [`wikimedia/wikipedia`](https://huggingface.co/datasets/wikimedia/wikipedia)
+- [`eljuanina/VotingBooklets-v1`](https://huggingface.co/datasets/eljuanina/VotingBooklets-v1)
+- [`ZurichNLP/SwissGov-RSD`](https://huggingface.co/datasets/ZurichNLP/SwissGov-RSD)
 
-Plain-text prefixes only:
+`eljuanina/VotingBooklets-Diamond-v1` is held-out evaluation-only and is therefore not listed in the
+training dataset metadata.
 
-- query: `query: `
-- document: `passage: `
+Actual training set: **31,483 triplets**.
+
+## Internal retrieval diagnostics
+
+These metrics are computed by reloading the **saved standalone Sentence Transformers model**.
+They are internal development diagnostics, not MTEB/MMTEB benchmark claims.
+
+| Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
+|---|---:|---:|---:|---:|---:|
+| Swiss monolingual | 1024 | 80.83% | 92.50% | 0.8651 | 0.8462 |
+| Swiss monolingual | 512 | 79.17% | 92.08% | 0.8570 | 0.8366 |
+| Swiss cross-lingual | 1024 | 30.00% | 96.67% | 0.6612 | 0.5598 |
+| Swiss cross-lingual | 512 | 29.38% | 96.88% | 0.6584 | 0.5556 |
+
+Monolingual evaluation queries: **480**  
+Cross-lingual evaluation queries: **480**
+
+## Router diagnostics
+
+The held-out routing diagnostics are saved in `router_diagnostics.json`.
+
+Recorded shared-expert Top-k selections in the held-out diagnostic:
+**0**.
+
+The declared language is used only as weak auxiliary supervision during MoE-head training;
+no language ID is required at inference.
 
-Inherited LLM chat templates are not part of the retrieval input pipeline.
+## Limitations
 
+- Sparse routing is currently in the embedding projection head; the Apertus backbone remains dense.
+- Wikipedia title→paragraph pairs are pseudo-retrieval supervision.
+- Parallel passages provide semantic alignment rather than natural search queries.
+- External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
+- The MoE head was trained on pooled representations up to
+  256 tokens. The packaged parent retains
+  1024-token inference capability, but quality beyond
+  the MoE training context length has not yet been separately validated.
 
-## Experimental note
+## License and attribution
 
-This repository is an experimental Language-MoE head artifact, not yet a drop-in
-Sentence Transformers release. `language_moe_config.json` pins the dense initializer
-revision and `language_moe_head.pt` contains the learned router/expert parameters.
+Apertus is developed by the Swiss AI Initiative. This is an independent retrieval/MoE
+adaptation and not an official Swiss AI Initiative embedding release.

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "ApertusModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float16",
+  "eos_token_id": 2,
+  "hidden_act": "xielu",
+  "hidden_dropout": 0.0,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "is_causal": false,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "apertus",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 20,
+  "num_key_value_heads": 4,
+  "pad_token_id": 3,
+  "post_norm": false,
+  "qk_norm": true,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 500000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 131072
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "__version__": {
+    "pytorch": "2.10.0+cu128",
+    "sentence_transformers": "5.6.1",
+    "transformers": "5.14.1"
+  },
+  "default_prompt_name": null,
+  "model_type": "SentenceTransformer",
+  "prompts": {
+    "document": "passage: ",
+    "query": "query: "
+  },
+  "similarity_fn_name": "dot"
+}

language_moe.py

@@ -0,0 +1,158 @@
+from typing import Any
+
+import torch
+from torch import nn
+from sentence_transformers.base.modules import Module
+
+
+class LanguageMoE(Module):
+    config_keys = [
+        "hidden_size",
+        "embedding_dim",
+        "expert_names",
+        "top_k",
+        "temperature",
+    ]
+
+    def __init__(
+        self,
+        hidden_size: int,
+        embedding_dim: int,
+        expert_names: list[str],
+        top_k: int = 2,
+        temperature: float = 1.0,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+        self.hidden_size = int(hidden_size)
+        self.embedding_dim = int(embedding_dim)
+        self.expert_names = list(expert_names)
+        self.top_k = int(top_k)
+        self.temperature = float(temperature)
+
+        assert 1 <= self.top_k <= len(self.expert_names)
+
+        self.router = nn.Linear(
+            self.hidden_size,
+            len(self.expert_names),
+            bias=True,
+        )
+        self.experts = nn.ModuleList([
+            nn.Linear(
+                self.hidden_size,
+                self.embedding_dim,
+                bias=False,
+            )
+            for _ in self.expert_names
+        ])
+
+        self.last_router_probs = None
+        self.last_top_indices = None
+
+    def route(self, x: torch.Tensor):
+        logits = self.router(x) / self.temperature
+        probs = torch.softmax(logits, dim=-1)
+
+        top_probs, top_idx = torch.topk(
+            probs,
+            k=self.top_k,
+            dim=-1,
+        )
+        gates = top_probs / top_probs.sum(
+            dim=-1,
+            keepdim=True,
+        ).clamp_min(1e-12)
+
+        return logits, probs, top_idx, gates
+
+    def project_sparse(
+        self,
+        x: torch.Tensor,
+        top_idx: torch.Tensor,
+        gates: torch.Tensor,
+    ) -> torch.Tensor:
+        output = x.new_zeros(
+            (x.shape[0], self.embedding_dim)
+        )
+
+        for expert_id, expert in enumerate(self.experts):
+            selected = top_idx.eq(expert_id)
+            rows, slots = selected.nonzero(as_tuple=True)
+
+            if rows.numel() == 0:
+                continue
+
+            expert_output = expert(x[rows])
+            expert_gate = gates[rows, slots].unsqueeze(-1)
+            output.index_add_(
+                0,
+                rows,
+                expert_output * expert_gate,
+            )
+
+        return output
+
+    def forward(
+        self,
+        features: dict[str, torch.Tensor | Any],
+        **kwargs,
+    ) -> dict[str, torch.Tensor | Any]:
+        x = features["sentence_embedding"]
+        _, probs, top_idx, gates = self.route(x)
+
+        features["sentence_embedding"] = self.project_sparse(
+            x,
+            top_idx,
+            gates,
+        )
+
+        self.last_router_probs = probs.detach()
+        self.last_top_indices = top_idx.detach()
+
+        return features
+
+    def get_embedding_dimension(self) -> int:
+        return self.embedding_dim
+
+    def save(
+        self,
+        output_path: str,
+        *args,
+        safe_serialization: bool = True,
+        **kwargs,
+    ) -> None:
+        self.save_config(output_path)
+        self.save_torch_weights(
+            output_path,
+            safe_serialization=safe_serialization,
+        )
+
+    @classmethod
+    def load(
+        cls,
+        model_name_or_path: str,
+        subfolder: str = "",
+        token: bool | str | None = None,
+        cache_folder: str | None = None,
+        revision: str | None = None,
+        local_files_only: bool = False,
+        **kwargs,
+    ):
+        config = cls.load_config(
+            model_name_or_path,
+            subfolder=subfolder,
+            token=token,
+            cache_folder=cache_folder,
+            revision=revision,
+            local_files_only=local_files_only,
+        )
+        model = cls(**config)
+        return cls.load_torch_weights(
+            model_name_or_path,
+            subfolder=subfolder,
+            token=token,
+            cache_folder=cache_folder,
+            revision=revision,
+            local_files_only=local_files_only,
+            model=model,
+        )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:210da7bac528f2eb9c3d5792ff4a6d5c8fb269ea5ad8b9f2797cf6e981a805fd
+size 876727640

model_parameters.json

@@ -1,10 +1,19 @@
 {
   "experiment_mode": "moe_head",
+  "architecture": "dense_backbone_language_moe_head",
   "source_model": "swiss-ai/Apertus-v1.1-0.5B",
+  "upstream_model": "swiss-ai/Apertus-8B-2509",
+  "apertus_version": "1.1",
   "base_model": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
   "base_revision": "a6f4b025ee825eec8be04d1132c9e24f20841778",
   "hidden_size": 1024,
   "embedding_dimension": 1024,
+  "matryoshka_dimensions": [
+    1024,
+    768,
+    512,
+    256
+  ],
   "expert_names": [
     "de",
     "en",
@@ -14,12 +23,20 @@
     "gsw",
     "shared"
   ],
-  "top_k": 2,
-  "head_parameters": 7347207,
+  "num_experts": 7,
+  "experts_active_per_sentence": 2,
+  "router_parameters": 7175,
+  "expert_parameters_each": 1048576,
+  "language_moe_head_parameters": 7347207,
+  "active_head_parameters_per_sentence": 2104327,
   "base_parameters_including_old_dense": 439397928,
-  "old_dense_parameters": 1048576,
-  "effective_parameters": 445696559,
-  "effective_parameters_billions": 0.445696559,
+  "old_dense_projection_parameters": 1048576,
+  "final_parameters": 445696559,
+  "final_parameters_billions": 0.445696559,
+  "active_parameters_per_sentence": 440453679,
   "repository_name": "apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
-  "training_profile": "size_optimized"
+  "training_profile": "size_optimized",
+  "moe_training_max_seq_length": 256,
+  "inference_max_seq_length": 1024,
+  "sparse_projection_equivalence_max_abs_diff": 1.9073486328125e-06
 }

modules.json

@@ -0,0 +1,26 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_LanguageMoE",
+    "type": "language_moe.LanguageMoE"
+  },
+  {
+    "idx": 3,
+    "name": "3",
+    "path": "3_Normalize",
+    "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
+  }
+]

router_diagnostics.json

@@ -1,139 +1,174 @@
 {
-  "document:de": {
-    "top1:en": 4,
-    "top2:de": 86,
-    "top1:de": 75,
-    "top2:en": 37,
-    "top2:fr": 24,
-    "top1:it": 3,
-    "top2:gsw": 13,
-    "top1:fr": 94,
-    "top2:it": 4,
-    "top2:rm": 12
-  },
-  "document:en": {
-    "top1:de": 46,
-    "top2:en": 47,
-    "top1:fr": 58,
-    "top2:de": 62,
-    "top1:en": 25,
-    "top2:fr": 12,
-    "top2:it": 11,
-    "top1:it": 4,
-    "top2:rm": 1
-  },
-  "document:fr": {
-    "top1:en": 10,
-    "top2:fr": 23,
-    "top1:it": 6,
-    "top2:de": 57,
-    "top1:de": 25,
-    "top2:it": 39,
-    "top1:fr": 138,
-    "top2:en": 9,
-    "top2:rm": 50,
-    "top2:gsw": 1
-  },
-  "document:gsw": {
-    "top1:de": 67,
-    "top2:fr": 23,
-    "top1:fr": 12,
-    "top2:de": 12,
-    "top2:gsw": 41,
-    "top2:rm": 1,
-    "top1:it": 1,
-    "top2:en": 3
-  },
-  "document:it": {
-    "top1:it": 39,
-    "top2:de": 34,
-    "top1:en": 6,
-    "top2:it": 78,
-    "top1:fr": 111,
-    "top2:fr": 27,
-    "top1:de": 22,
-    "top2:en": 12,
-    "top2:rm": 28,
-    "top1:rm": 1
-  },
-  "document:rm": {
-    "top1:it": 10,
-    "top2:fr": 26,
-    "top1:de": 36,
-    "top2:it": 30,
-    "top1:fr": 91,
-    "top2:rm": 55,
-    "top1:rm": 4,
-    "top2:de": 21,
-    "top2:gsw": 5,
-    "top2:en": 4
-  },
-  "query:de": {
-    "top1:de": 83,
-    "top2:en": 35,
-    "top2:gsw": 22,
-    "top2:fr": 20,
-    "top1:it": 3,
-    "top2:de": 94,
-    "top2:it": 8,
-    "top1:en": 2,
-    "top1:fr": 102,
-    "top2:rm": 11
-  },
-  "query:en": {
-    "top1:de": 68,
-    "top2:en": 61,
-    "top2:fr": 15,
-    "top1:en": 13,
-    "top1:fr": 56,
-    "top2:de": 51,
-    "top2:it": 8,
-    "top1:it": 3,
-    "top2:gsw": 1,
-    "top2:rm": 4
-  },
-  "query:fr": {
-    "top1:fr": 126,
-    "top2:de": 33,
-    "top1:it": 5,
-    "top1:de": 58,
-    "top2:en": 25,
-    "top2:fr": 34,
-    "top1:en": 1,
-    "top2:gsw": 2,
-    "top2:it": 43,
-    "top2:rm": 53
-  },
-  "query:gsw": {
-    "top1:de": 75,
-    "top2:it": 4,
-    "top2:fr": 28,
-    "top2:gsw": 39,
-    "top1:fr": 2,
-    "top2:de": 4,
-    "top1:it": 3,
-    "top2:en": 5
-  },
-  "query:it": {
-    "top1:it": 31,
-    "top2:de": 36,
-    "top2:en": 21,
-    "top1:fr": 112,
-    "top2:it": 88,
-    "top1:de": 45,
-    "top2:fr": 13,
-    "top1:en": 2,
-    "top2:rm": 32
-  },
-  "query:rm": {
-    "top1:it": 9,
-    "top2:de": 18,
-    "top1:de": 55,
-    "top2:en": 9,
-    "top2:fr": 22,
-    "top2:rm": 76,
-    "top1:fr": 106,
-    "top2:it": 38,
-    "top2:gsw": 7
+  "expert_names": [
+    "de",
+    "en",
+    "fr",
+    "it",
+    "rm",
+    "gsw",
+    "shared"
+  ],
+  "top_k": 2,
+  "shared_expert_topk_selections": 0,
+  "by_role_and_declared_language": {
+    "document:de": {
+      "top1:en": 2,
+      "top2:de": 87,
+      "top1:de": 87,
+      "top2:en": 33,
+      "top2:fr": 35,
+      "top1:it": 2,
+      "top2:gsw": 15,
+      "top1:fr": 85,
+      "top2:it": 4,
+      "top2:rm": 2,
+      "n": 176,
+      "mean_router_entropy": 1.609473284090909
+    },
+    "document:en": {
+      "top1:de": 56,
+      "top2:en": 42,
+      "top2:fr": 17,
+      "top1:fr": 55,
+      "top2:de": 67,
+      "top1:en": 19,
+      "top2:it": 7,
+      "top1:it": 3,
+      "n": 133,
+      "mean_router_entropy": 1.6641771804511278
+    },
+    "document:fr": {
+      "top1:en": 6,
+      "top2:fr": 30,
+      "top1:it": 5,
+      "top2:de": 76,
+      "top1:de": 30,
+      "top1:fr": 138,
+      "top2:it": 32,
+      "top2:en": 6,
+      "top2:rm": 35,
+      "n": 179,
+      "mean_router_entropy": 1.5592967374301678
+    },
+    "document:gsw": {
+      "top1:de": 67,
+      "top2:fr": 22,
+      "top1:fr": 12,
+      "top2:de": 12,
+      "top2:gsw": 42,
+      "top2:rm": 1,
+      "top1:it": 1,
+      "top2:en": 2,
+      "top2:it": 1,
+      "n": 80,
+      "mean_router_entropy": 1.6445136875
+    },
+    "document:it": {
+      "top1:it": 35,
+      "top2:de": 43,
+      "top2:en": 11,
+      "top1:fr": 116,
+      "top2:it": 82,
+      "top1:de": 26,
+      "top2:fr": 23,
+      "top1:en": 2,
+      "top2:rm": 20,
+      "n": 179,
+      "mean_router_entropy": 1.6042385754189945
+    },
+    "document:rm": {
+      "top1:it": 6,
+      "top2:fr": 30,
+      "top1:de": 36,
+      "top1:fr": 93,
+      "top2:rm": 58,
+      "top1:rm": 5,
+      "top2:it": 21,
+      "top2:de": 26,
+      "top2:gsw": 3,
+      "top2:en": 3,
+      "top1:en": 1,
+      "n": 141,
+      "mean_router_entropy": 1.6795429858156028
+    },
+    "query:de": {
+      "top1:de": 86,
+      "top2:en": 27,
+      "top2:gsw": 20,
+      "top2:fr": 30,
+      "top1:it": 3,
+      "top2:de": 102,
+      "top2:it": 9,
+      "top1:en": 2,
+      "top1:fr": 99,
+      "top2:rm": 2,
+      "n": 190,
+      "mean_router_entropy": 1.6067308421052633
+    },
+    "query:en": {
+      "top1:de": 79,
+      "top2:en": 54,
+      "top2:fr": 22,
+      "top1:en": 6,
+      "top1:fr": 53,
+      "top2:de": 53,
+      "top2:it": 7,
+      "top1:it": 2,
+      "top2:rm": 3,
+      "top2:gsw": 1,
+      "n": 140,
+      "mean_router_entropy": 1.6650936142857142
+    },
+    "query:fr": {
+      "top1:de": 52,
+      "top2:fr": 37,
+      "top1:it": 5,
+      "top2:de": 75,
+      "top2:it": 23,
+      "top1:fr": 132,
+      "top1:en": 1,
+      "top2:en": 17,
+      "top2:gsw": 2,
+      "top2:rm": 36,
+      "n": 190,
+      "mean_router_entropy": 1.5682451
+    },
+    "query:gsw": {
+      "top1:de": 75,
+      "top2:it": 2,
+      "top2:fr": 33,
+      "top2:gsw": 36,
+      "top1:it": 3,
+      "top2:en": 5,
+      "top2:de": 4,
+      "top1:fr": 2,
+      "n": 80,
+      "mean_router_entropy": 1.5985225375
+    },
+    "query:it": {
+      "top1:it": 29,
+      "top2:de": 46,
+      "top2:en": 15,
+      "top1:fr": 115,
+      "top2:it": 87,
+      "top1:de": 46,
+      "top2:fr": 21,
+      "top2:rm": 21,
+      "n": 190,
+      "mean_router_entropy": 1.612266494736842
+    },
+    "query:rm": {
+      "top1:it": 9,
+      "top2:fr": 36,
+      "top1:de": 53,
+      "top2:en": 6,
+      "top1:fr": 108,
+      "top2:de": 23,
+      "top2:rm": 73,
+      "top2:gsw": 7,
+      "top2:it": 25,
+      "n": 170,
+      "mean_router_entropy": 1.654621305882353
+    }
   }
 }

sentence_bert_config.json

@@ -0,0 +1,10 @@
+{
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        }
+    },
+    "module_output_name": "token_embeddings"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be12f4375d655cc740864e3a9041bcddd8477942f209d9e7f27f6c8767162638
+size 17078368

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1024,
+  "pad_token": "[INST]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

language_moe_config.json → training_metadata.json

@@ -1,11 +1,14 @@
 {
-  "base_model": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
-  "base_revision": "a6f4b025ee825eec8be04d1132c9e24f20841778",
-  "hidden_size": 1024,
-  "embedding_dim": 1024,
-  "max_seq_length": 256,
-  "num_triplets": 31483,
-  "num_unique_prompted_texts": 56324,
+  "experiment_mode": "moe_head",
+  "source_model": "swiss-ai/Apertus-v1.1-0.5B",
+  "dense_parent": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
+  "dense_parent_revision": "a6f4b025ee825eec8be04d1132c9e24f20841778",
+  "backbone_frozen_during_moe_training": true,
+  "training_profile": "size_optimized",
+  "training_triplets": 31483,
+  "unique_prompted_texts": 56324,
+  "moe_training_max_seq_length": 256,
+  "inference_max_seq_length": 1024,
   "expert_names": [
     "de",
     "en",
@@ -16,8 +19,6 @@
     "shared"
   ],
   "top_k": 2,
-  "experiment_mode": "moe_head",
-  "training_profile": "size_optimized",
   "router_temperature": 1.0,
   "language_loss_weight": 0.05,
   "balance_loss_weight": 0.01,
@@ -39,26 +40,26 @@
   "training_history": [
     {
       "epoch": 1,
-      "loss": 1.5221745264334756,
-      "retrieval_loss": 1.437075538713424,
-      "language_loss": 1.7009963930630294,
-      "balance_loss": 0.004917386882641685,
+      "loss": 1.5206839119801756,
+      "retrieval_loss": 1.435533842102426,
+      "language_loss": 1.7019310056186112,
+      "balance_loss": 0.00535158973679111,
       "lr": 0.00015
     },
     {
       "epoch": 2,
-      "loss": 1.3965658281670241,
-      "retrieval_loss": 1.3187617298032417,
-      "language_loss": 1.5544935601656553,
-      "balance_loss": 0.00794149145911463,
+      "loss": 1.4084786567531649,
+      "retrieval_loss": 1.3307430470576052,
+      "language_loss": 1.5530201450723116,
+      "balance_loss": 0.008459957545531577,
       "lr": 5.0000000000000057e-05
     },
     {
       "epoch": 3,
-      "loss": 1.3498401231453068,
-      "retrieval_loss": 1.2746383928861775,
-      "language_loss": 1.5026149945180924,
-      "balance_loss": 0.007097912257628851,
+      "loss": 1.3571588367712302,
+      "retrieval_loss": 1.281926788267542,
+      "language_loss": 1.503104266573171,
+      "balance_loss": 0.007683004374753256,
       "lr": 0.0
     }
   ]