Hugging Face's logo

andreasmartin
/
apertus-v1.1-swiss-embed-0.4b-bidir-langmoe

Sentence Similarity
sentence-transformers
Safetensors
apertus
embeddings
retrieval
multilingual
swiss
apertus-1.1
bidirectional
matryoshka
Mixture of Experts
language-moe
sparse-routing
Model card Files
xet
 Community
andreasmartin commited on
Commit
a436d22
·
verified ·
1 Parent(s): 8a0a543

Update standalone Apertus embedding model: moe_head

Browse files
Files changed (8) hide show
  1. 2_LanguageMoE/model.safetensors +1 -1
  2. README.md +26 -21
  3. config_sentence_transformers.json +1 -1
  4. data_manifest.json +67 -0
  5. model.safetensors +1 -1
  6. model_parameters.json +4 -4
  7. router_diagnostics.json +106 -109
  8. training_metadata.json +43 -19
2_LanguageMoE/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c7544c30354147275e494c847a59ff011e09ef467210ff64c6bfd7d11b941caf
3
  size 14695206
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bd99a0d48718bd7a26a7cd194aa21f5029b49a96ad10b108066a80183b6f4186
3
  size 14695206
README.md CHANGED
@@ -45,18 +45,18 @@ head**.
45
 
46
  ```text
47
  plain retrieval text
48
- ↓
49
  bidirectional Apertus encoder
50
- ↓
51
  mean pooling
52
- ↓
53
  Language-MoE router
54
- ↓
55
  Top-2 of:
56
  DE / EN / FR / IT / RM / GSW / shared
57
- ↓
58
  1024d projection
59
- ↓
60
  L2 normalization
61
  ```
62
 
@@ -72,9 +72,9 @@ L2 normalization
72
  | Active experts / sentence | 2 |
73
  | Native embedding dimension | **1024** |
74
  | Matryoshka dimensions | `[1024, 768, 512, 256]` |
75
- | MoE training max length | **256 tokens** |
76
  | Inference max length | **1024 tokens** |
77
- | Sparse/reference max abs diff | `1.907e-06` |
78
 
79
  ## Sentence Transformers usage
80
 
@@ -85,17 +85,17 @@ L2 normalization
85
  from sentence_transformers import SentenceTransformer
86
 
87
  model = SentenceTransformer(
88
- "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
89
- trust_remote_code=True,
90
  )
91
 
92
  queries = model.encode_query([
93
- "Welche Aufgaben hat der Bundesrat?"
94
  ])
95
 
96
  documents = model.encode_document([
97
- "Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
98
- "Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
99
  ])
100
 
101
  scores = model.similarity(queries, documents)
@@ -106,8 +106,8 @@ For a compact Matryoshka representation:
106
 
107
  ```python
108
  embeddings = model.encode_query(
109
- texts,
110
- truncate_dim=512,
111
  )
112
  ```
113
 
@@ -124,7 +124,7 @@ No chat template is used.
124
  |---|---|
125
  | Original source | [`swiss-ai/Apertus-v1.1-0.5B`](https://huggingface.co/swiss-ai/Apertus-v1.1-0.5B) |
126
  | Dense embedding parent | [`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir) |
127
- | Dense parent revision | `a6f4b025ee825eec8be04d1132c9e24f20841778` |
128
  | Apertus lineage | [`swiss-ai/Apertus-8B-2509`](https://huggingface.co/swiss-ai/Apertus-8B-2509) |
129
 
130
  The repository contains the complete transformer/tokenizer/pooling/MoE/normalization pipeline.
@@ -149,6 +149,11 @@ training dataset metadata.
149
 
150
  Actual training set: **31,483 triplets**.
151
 
 
 
 
 
 
152
  ## Internal retrieval diagnostics
153
 
154
  These metrics are computed by reloading the **saved standalone Sentence Transformers model**.
@@ -156,10 +161,10 @@ They are internal development diagnostics, not MTEB/MMTEB benchmark claims.
156
 
157
  | Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
158
  |---|---:|---:|---:|---:|---:|
159
- | Swiss monolingual | 1024 | 80.83% | 92.50% | 0.8651 | 0.8462 |
160
- | Swiss monolingual | 512 | 79.17% | 92.08% | 0.8570 | 0.8366 |
161
- | Swiss cross-lingual | 1024 | 30.00% | 96.67% | 0.6612 | 0.5598 |
162
- | Swiss cross-lingual | 512 | 29.38% | 96.88% | 0.6584 | 0.5556 |
163
 
164
  Monolingual evaluation queries: **480**
165
  Cross-lingual evaluation queries: **480**
@@ -181,7 +186,7 @@ no language ID is required at inference.
181
  - Parallel passages provide semantic alignment rather than natural search queries.
182
  - External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
183
  - The MoE head was trained on pooled representations up to
184
- 256 tokens. The packaged parent retains
185
  1024-token inference capability, but quality beyond
186
  the MoE training context length has not yet been separately validated.
187
 
 
45
 
46
  ```text
47
  plain retrieval text
48
+ ↓
49
  bidirectional Apertus encoder
50
+ ↓
51
  mean pooling
52
+ ↓
53
  Language-MoE router
54
+ ↓
55
  Top-2 of:
56
  DE / EN / FR / IT / RM / GSW / shared
57
+ ↓
58
  1024d projection
59
+ ↓
60
  L2 normalization
61
  ```
62
 
 
72
  | Active experts / sentence | 2 |
73
  | Native embedding dimension | **1024** |
74
  | Matryoshka dimensions | `[1024, 768, 512, 256]` |
75
+ | MoE training max length | **512 tokens** |
76
  | Inference max length | **1024 tokens** |
77
+ | Sparse/reference max abs diff | `1.431e-06` |
78
 
79
  ## Sentence Transformers usage
80
 
 
85
  from sentence_transformers import SentenceTransformer
86
 
87
  model = SentenceTransformer(
88
+ "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
89
+ trust_remote_code=True,
90
  )
91
 
92
  queries = model.encode_query([
93
+ "Welche Aufgaben hat der Bundesrat?"
94
  ])
95
 
96
  documents = model.encode_document([
97
+ "Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
98
+ "Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
99
  ])
100
 
101
  scores = model.similarity(queries, documents)
 
106
 
107
  ```python
108
  embeddings = model.encode_query(
109
+ texts,
110
+ truncate_dim=512,
111
  )
112
  ```
113
 
 
124
  |---|---|
125
  | Original source | [`swiss-ai/Apertus-v1.1-0.5B`](https://huggingface.co/swiss-ai/Apertus-v1.1-0.5B) |
126
  | Dense embedding parent | [`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir) |
127
+ | Dense parent revision | `875c1f99d7b20e261ea43b3586fdde5f8d5648e5` |
128
  | Apertus lineage | [`swiss-ai/Apertus-8B-2509`](https://huggingface.co/swiss-ai/Apertus-8B-2509) |
129
 
130
  The repository contains the complete transformer/tokenizer/pooling/MoE/normalization pipeline.
 
149
 
150
  Actual training set: **31,483 triplets**.
151
 
152
+ The MoE stage consumes the shared `train_moe.jsonl` generated together with the Dense
153
+ `train.jsonl`; triplets and negatives are identical. The released `data_manifest.json` records
154
+ the exact Hugging Face dataset commit SHAs, recipe signature and SHA256 hashes of the processed
155
+ files.
156
+
157
  ## Internal retrieval diagnostics
158
 
159
  These metrics are computed by reloading the **saved standalone Sentence Transformers model**.
 
161
 
162
  | Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
163
  |---|---:|---:|---:|---:|---:|
164
+ | Swiss monolingual | 1024 | 82.08% | 91.88% | 0.8678 | 0.8517 |
165
+ | Swiss monolingual | 512 | 80.00% | 91.88% | 0.8593 | 0.8404 |
166
+ | Swiss cross-lingual | 1024 | 30.63% | 96.04% | 0.6627 | 0.5635 |
167
+ | Swiss cross-lingual | 512 | 30.42% | 96.04% | 0.6602 | 0.5605 |
168
 
169
  Monolingual evaluation queries: **480**
170
  Cross-lingual evaluation queries: **480**
 
186
  - Parallel passages provide semantic alignment rather than natural search queries.
187
  - External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
188
  - The MoE head was trained on pooled representations up to
189
+ 512 tokens. The packaged parent retains
190
  1024-token inference capability, but quality beyond
191
  the MoE training context length has not yet been separately validated.
192
 
config_sentence_transformers.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "__version__": {
3
- "pytorch": "2.10.0+cu128",
4
  "sentence_transformers": "5.6.1",
5
  "transformers": "5.14.1"
6
  },
 
1
  {
2
  "__version__": {
3
+ "pytorch": "2.8.0+cu128",
4
  "sentence_transformers": "5.6.1",
5
  "transformers": "5.14.1"
6
  },
data_manifest.json ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 2,
3
+ "created_at_utc": "2026-08-21T19:56:58.229241+00:00",
4
+ "provider_first_built_on": "Lightning AI",
5
+ "shared_data_directory": "/teamspace/studios/this_studio/apertus_shared/swiss-retrieval-data",
6
+ "recipe": {
7
+ "schema_version": 2,
8
+ "smoke_test": false,
9
+ "seed": 42,
10
+ "wiki_date": "20231101",
11
+ "swiss_mono_targets": {
12
+ "de": 6000,
13
+ "en": 4000,
14
+ "fr": 5000,
15
+ "it": 4000,
16
+ "rm": 2500,
17
+ "als": 2500
18
+ },
19
+ "mono_eval_per_lang": 80,
20
+ "voting_train_per_direction": 500,
21
+ "diamond_eval_per_direction": 30,
22
+ "swissgov_train_per_direction": 150,
23
+ "swissgov_eval_per_direction": 20,
24
+ "preservation_targets": {
25
+ "es": 200,
26
+ "pt": 200,
27
+ "nl": 150,
28
+ "pl": 150,
29
+ "tr": 150,
30
+ "ar": 150,
31
+ "hi": 150,
32
+ "zh": 150,
33
+ "sw": 150
34
+ },
35
+ "min_doc_chars": 220,
36
+ "max_doc_chars": 1800,
37
+ "min_parallel_chars": 60,
38
+ "max_parallel_chars": 1600
39
+ },
40
+ "recipe_signature": "fd8f7d87d50416d4d732c68dfffd2ce3de5a58aac879ccb677185d06186c400e",
41
+ "sources": {
42
+ "wikimedia/wikipedia": {
43
+ "url": "https://huggingface.co/datasets/wikimedia/wikipedia",
44
+ "revision": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa"
45
+ },
46
+ "ZurichNLP/SwissGov-RSD": {
47
+ "url": "https://huggingface.co/datasets/ZurichNLP/SwissGov-RSD",
48
+ "revision": "f16aa0536811b8c0cf975fede21eec5996e4b3c1"
49
+ },
50
+ "eljuanina/VotingBooklets-v1": {
51
+ "url": "https://huggingface.co/datasets/eljuanina/VotingBooklets-v1",
52
+ "revision": "8db11c674d6993aee084fe65ed3ab6d9ee102fd7"
53
+ },
54
+ "eljuanina/VotingBooklets-Diamond-v1": {
55
+ "url": "https://huggingface.co/datasets/eljuanina/VotingBooklets-Diamond-v1",
56
+ "revision": "476c96e74a72b65122cf6fc01943be125f7f3324"
57
+ }
58
+ },
59
+ "processed_file_sha256": {
60
+ "train.jsonl": "d84c3dc14244c746deed98cee9cd5332e16f483b3cab039a53fe7c8c994dfcaf",
61
+ "train_moe.jsonl": "287bee32718f74d333f48e94c38e5150ed70921f47e15871940c83ce7e13d22a",
62
+ "eval_mono.jsonl": "b974eebcc3b3c68b443eec04315ab96ff3d5c4cf7fc0a54785fc50e1597ecc27",
63
+ "eval_cross.jsonl": "82ba1f01c9099a39b30bec0b33dbc48b5bf6998973f45387d389a8a951144f19",
64
+ "data_stats.json": "ee05572b590907e92ba753799994a8914d53dbb1962e16482ab3c42b78dd2e30"
65
+ },
66
+ "train_triplets_identical_between_dense_and_moe": true
67
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:210da7bac528f2eb9c3d5792ff4a6d5c8fb269ea5ad8b9f2797cf6e981a805fd
3
  size 876727640
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:208c3783c0345d4defa0a6944b6d35d401a2179e41f03b555e40488de99c8213
3
  size 876727640
model_parameters.json CHANGED
@@ -5,7 +5,7 @@
5
  "upstream_model": "swiss-ai/Apertus-8B-2509",
6
  "apertus_version": "1.1",
7
  "base_model": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
8
- "base_revision": "a6f4b025ee825eec8be04d1132c9e24f20841778",
9
  "hidden_size": 1024,
10
  "embedding_dimension": 1024,
11
  "matryoshka_dimensions": [
@@ -35,8 +35,8 @@
35
  "final_parameters_billions": 0.445696559,
36
  "active_parameters_per_sentence": 440453679,
37
  "repository_name": "apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
38
- "training_profile": "size_optimized",
39
- "moe_training_max_seq_length": 256,
40
  "inference_max_seq_length": 1024,
41
- "sparse_projection_equivalence_max_abs_diff": 1.9073486328125e-06
42
  }
 
5
  "upstream_model": "swiss-ai/Apertus-8B-2509",
6
  "apertus_version": "1.1",
7
  "base_model": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
8
+ "base_revision": "875c1f99d7b20e261ea43b3586fdde5f8d5648e5",
9
  "hidden_size": 1024,
10
  "embedding_dimension": 1024,
11
  "matryoshka_dimensions": [
 
35
  "final_parameters_billions": 0.445696559,
36
  "active_parameters_per_sentence": 440453679,
37
  "repository_name": "apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
38
+ "training_profile": "quality",
39
+ "moe_training_max_seq_length": 512,
40
  "inference_max_seq_length": 1024,
41
+ "sparse_projection_equivalence_max_abs_diff": 1.430511474609375e-06
42
  }
router_diagnostics.json CHANGED
@@ -12,163 +12,160 @@
12
  "shared_expert_topk_selections": 0,
13
  "by_role_and_declared_language": {
14
  "document:de": {
15
- "top1:en": 2,
16
- "top2:de": 87,
17
- "top1:de": 87,
18
- "top2:en": 33,
19
- "top2:fr": 35,
20
- "top1:it": 2,
21
- "top2:gsw": 15,
22
- "top1:fr": 85,
23
- "top2:it": 4,
24
- "top2:rm": 2,
25
  "n": 176,
26
- "mean_router_entropy": 1.609473284090909
27
  },
28
  "document:en": {
29
- "top1:de": 56,
30
- "top2:en": 42,
31
- "top2:fr": 17,
32
- "top1:fr": 55,
33
- "top2:de": 67,
34
- "top1:en": 19,
35
- "top2:it": 7,
36
- "top1:it": 3,
37
  "n": 133,
38
- "mean_router_entropy": 1.6641771804511278
39
  },
40
  "document:fr": {
41
- "top1:en": 6,
42
- "top2:fr": 30,
43
- "top1:it": 5,
44
- "top2:de": 76,
45
- "top1:de": 30,
46
- "top1:fr": 138,
47
- "top2:it": 32,
48
- "top2:en": 6,
49
- "top2:rm": 35,
50
  "n": 179,
51
- "mean_router_entropy": 1.5592967374301678
52
  },
53
  "document:gsw": {
54
- "top1:de": 67,
55
- "top2:fr": 22,
56
- "top1:fr": 12,
57
- "top2:de": 12,
58
- "top2:gsw": 42,
 
59
  "top2:rm": 1,
60
  "top1:it": 1,
61
- "top2:en": 2,
62
- "top2:it": 1,
63
  "n": 80,
64
- "mean_router_entropy": 1.6445136875
65
  },
66
  "document:it": {
67
- "top1:it": 35,
68
- "top2:de": 43,
69
- "top2:en": 11,
70
- "top1:fr": 116,
71
- "top2:it": 82,
72
- "top1:de": 26,
73
- "top2:fr": 23,
74
- "top1:en": 2,
75
- "top2:rm": 20,
76
  "n": 179,
77
- "mean_router_entropy": 1.6042385754189945
78
  },
79
  "document:rm": {
80
- "top1:it": 6,
81
- "top2:fr": 30,
82
- "top1:de": 36,
83
- "top1:fr": 93,
84
- "top2:rm": 58,
85
- "top1:rm": 5,
86
  "top2:it": 21,
87
- "top2:de": 26,
 
88
  "top2:gsw": 3,
89
  "top2:en": 3,
90
  "top1:en": 1,
91
  "n": 141,
92
- "mean_router_entropy": 1.6795429858156028
93
  },
94
  "query:de": {
95
- "top1:de": 86,
96
- "top2:en": 27,
97
- "top2:gsw": 20,
98
- "top2:fr": 30,
99
- "top1:it": 3,
100
- "top2:de": 102,
101
- "top2:it": 9,
102
- "top1:en": 2,
103
- "top1:fr": 99,
104
- "top2:rm": 2,
105
  "n": 190,
106
- "mean_router_entropy": 1.6067308421052633
107
  },
108
  "query:en": {
109
- "top1:de": 79,
110
- "top2:en": 54,
111
- "top2:fr": 22,
112
- "top1:en": 6,
113
- "top1:fr": 53,
114
- "top2:de": 53,
115
- "top2:it": 7,
116
- "top1:it": 2,
117
  "top2:rm": 3,
 
118
  "top2:gsw": 1,
119
  "n": 140,
120
- "mean_router_entropy": 1.6650936142857142
121
  },
122
  "query:fr": {
123
- "top1:de": 52,
124
- "top2:fr": 37,
125
- "top1:it": 5,
126
- "top2:de": 75,
127
- "top2:it": 23,
128
- "top1:fr": 132,
129
- "top1:en": 1,
130
- "top2:en": 17,
131
  "top2:gsw": 2,
132
- "top2:rm": 36,
133
  "n": 190,
134
- "mean_router_entropy": 1.5682451
135
  },
136
  "query:gsw": {
137
  "top1:de": 75,
138
- "top2:it": 2,
139
- "top2:fr": 33,
140
  "top2:gsw": 36,
 
 
141
  "top1:it": 3,
142
- "top2:en": 5,
143
- "top2:de": 4,
144
  "top1:fr": 2,
145
  "n": 80,
146
- "mean_router_entropy": 1.5985225375
147
  },
148
  "query:it": {
149
- "top1:it": 29,
150
  "top2:de": 46,
151
- "top2:en": 15,
152
- "top1:fr": 115,
153
- "top2:it": 87,
154
- "top1:de": 46,
155
- "top2:fr": 21,
156
- "top2:rm": 21,
 
157
  "n": 190,
158
- "mean_router_entropy": 1.612266494736842
159
  },
160
  "query:rm": {
161
- "top1:it": 9,
162
- "top2:fr": 36,
163
- "top1:de": 53,
164
- "top2:en": 6,
165
- "top1:fr": 108,
166
- "top2:de": 23,
167
- "top2:rm": 73,
168
- "top2:gsw": 7,
169
- "top2:it": 25,
170
  "n": 170,
171
- "mean_router_entropy": 1.654621305882353
172
  }
173
  }
174
  }
 
12
  "shared_expert_topk_selections": 0,
13
  "by_role_and_declared_language": {
14
  "document:de": {
15
+ "top1:en": 4,
16
+ "top2:de": 49,
17
+ "top1:de": 96,
18
+ "top2:en": 25,
19
+ "top2:fr": 65,
20
+ "top1:it": 38,
21
+ "top2:gsw": 11,
22
+ "top1:fr": 38,
23
+ "top2:it": 26,
 
24
  "n": 176,
25
+ "mean_router_entropy": 1.685270244318182
26
  },
27
  "document:en": {
28
+ "top1:en": 27,
29
+ "top2:de": 59,
30
+ "top1:de": 50,
31
+ "top2:fr": 31,
32
+ "top1:fr": 46,
33
+ "top2:it": 13,
34
+ "top2:en": 30,
35
+ "top1:it": 10,
36
  "n": 133,
37
+ "mean_router_entropy": 1.705111112781955
38
  },
39
  "document:fr": {
40
+ "top1:en": 7,
41
+ "top2:fr": 64,
42
+ "top1:it": 45,
43
+ "top2:de": 52,
44
+ "top1:de": 27,
45
+ "top2:it": 56,
46
+ "top1:fr": 100,
47
+ "top2:en": 7,
 
48
  "n": 179,
49
+ "mean_router_entropy": 1.6696601396648045
50
  },
51
  "document:gsw": {
52
+ "top1:de": 71,
53
+ "top2:fr": 28,
54
+ "top2:gsw": 38,
55
+ "top1:fr": 8,
56
+ "top2:de": 8,
57
+ "top2:it": 2,
58
  "top2:rm": 1,
59
  "top1:it": 1,
60
+ "top2:en": 3,
 
61
  "n": 80,
62
+ "mean_router_entropy": 1.6836649625
63
  },
64
  "document:it": {
65
+ "top1:it": 127,
66
+ "top2:de": 37,
67
+ "top2:en": 13,
68
+ "top2:fr": 109,
69
+ "top1:de": 25,
70
+ "top2:it": 19,
71
+ "top1:fr": 23,
72
+ "top1:en": 4,
73
+ "top2:rm": 1,
74
  "n": 179,
75
+ "mean_router_entropy": 1.6558301787709497
76
  },
77
  "document:rm": {
78
+ "top1:it": 78,
79
+ "top2:de": 21,
80
+ "top1:de": 39,
81
+ "top2:fr": 61,
82
+ "top1:fr": 19,
 
83
  "top2:it": 21,
84
+ "top2:rm": 32,
85
+ "top1:rm": 4,
86
  "top2:gsw": 3,
87
  "top2:en": 3,
88
  "top1:en": 1,
89
  "n": 141,
90
+ "mean_router_entropy": 1.716398134751773
91
  },
92
  "query:de": {
93
+ "top1:en": 4,
94
+ "top2:de": 59,
95
+ "top1:de": 108,
96
+ "top2:en": 31,
97
+ "top2:gsw": 22,
98
+ "top2:fr": 46,
99
+ "top1:it": 26,
100
+ "top2:it": 32,
101
+ "top1:fr": 52,
 
102
  "n": 190,
103
+ "mean_router_entropy": 1.6838624473684212
104
  },
105
  "query:en": {
106
+ "top1:en": 17,
107
+ "top2:de": 49,
108
+ "top1:de": 73,
109
+ "top2:en": 47,
110
+ "top2:fr": 26,
111
+ "top1:it": 7,
112
+ "top2:it": 14,
 
113
  "top2:rm": 3,
114
+ "top1:fr": 43,
115
  "top2:gsw": 1,
116
  "n": 140,
117
+ "mean_router_entropy": 1.7038726285714285
118
  },
119
  "query:fr": {
120
+ "top1:de": 51,
121
+ "top2:fr": 64,
122
+ "top1:it": 47,
123
+ "top2:de": 39,
124
+ "top2:en": 21,
125
+ "top1:fr": 90,
126
+ "top1:en": 2,
 
127
  "top2:gsw": 2,
128
+ "top2:it": 64,
129
  "n": 190,
130
+ "mean_router_entropy": 1.6880721684210527
131
  },
132
  "query:gsw": {
133
  "top1:de": 75,
134
+ "top2:fr": 27,
 
135
  "top2:gsw": 36,
136
+ "top2:it": 3,
137
+ "top2:en": 9,
138
  "top1:it": 3,
139
+ "top2:de": 5,
 
140
  "top1:fr": 2,
141
  "n": 80,
142
+ "mean_router_entropy": 1.6277479750000001
143
  },
144
  "query:it": {
145
+ "top1:it": 125,
146
  "top2:de": 46,
147
+ "top1:en": 2,
148
+ "top2:it": 30,
149
+ "top2:fr": 92,
150
+ "top1:de": 50,
151
+ "top2:en": 17,
152
+ "top1:fr": 13,
153
+ "top2:rm": 5,
154
  "n": 190,
155
+ "mean_router_entropy": 1.6620954894736841
156
  },
157
  "query:rm": {
158
+ "top1:it": 106,
159
+ "top2:de": 22,
160
+ "top1:de": 58,
161
+ "top2:en": 9,
162
+ "top2:it": 20,
163
+ "top2:rm": 49,
164
+ "top2:fr": 62,
165
+ "top2:gsw": 8,
166
+ "top1:fr": 6,
167
  "n": 170,
168
+ "mean_router_entropy": 1.6873487411764707
169
  }
170
  }
171
  }
training_metadata.json CHANGED
@@ -2,12 +2,20 @@
2
  "experiment_mode": "moe_head",
3
  "source_model": "swiss-ai/Apertus-v1.1-0.5B",
4
  "dense_parent": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
5
- "dense_parent_revision": "a6f4b025ee825eec8be04d1132c9e24f20841778",
6
  "backbone_frozen_during_moe_training": true,
7
- "training_profile": "size_optimized",
 
 
 
 
 
 
 
 
8
  "training_triplets": 31483,
9
  "unique_prompted_texts": 56324,
10
- "moe_training_max_seq_length": 256,
11
  "inference_max_seq_length": 1024,
12
  "expert_names": [
13
  "de",
@@ -23,8 +31,8 @@
23
  "language_loss_weight": 0.05,
24
  "balance_loss_weight": 0.01,
25
  "head_learning_rate": 0.0002,
26
- "head_batch_size": 512,
27
- "head_epochs": 3,
28
  "matryoshka_dims": [
29
  1024,
30
  768,
@@ -40,26 +48,42 @@
40
  "training_history": [
41
  {
42
  "epoch": 1,
43
- "loss": 1.5206839119801756,
44
- "retrieval_loss": 1.435533842102426,
45
- "language_loss": 1.7019310056186112,
46
- "balance_loss": 0.00535158973679111,
47
- "lr": 0.00015
48
  },
49
  {
50
  "epoch": 2,
51
- "loss": 1.4084786567531649,
52
- "retrieval_loss": 1.3307430470576052,
53
- "language_loss": 1.5530201450723116,
54
- "balance_loss": 0.008459957545531577,
55
- "lr": 5.0000000000000057e-05
56
  },
57
  {
58
  "epoch": 3,
59
- "loss": 1.3571588367712302,
60
- "retrieval_loss": 1.281926788267542,
61
- "language_loss": 1.503104266573171,
62
- "balance_loss": 0.007683004374753256,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  "lr": 0.0
64
  }
65
  ]
 
2
  "experiment_mode": "moe_head",
3
  "source_model": "swiss-ai/Apertus-v1.1-0.5B",
4
  "dense_parent": "andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir",
5
+ "dense_parent_revision": "875c1f99d7b20e261ea43b3586fdde5f8d5648e5",
6
  "backbone_frozen_during_moe_training": true,
7
+ "training_profile": "quality",
8
+ "data_recipe_signature": "fd8f7d87d50416d4d732c68dfffd2ce3de5a58aac879ccb677185d06186c400e",
9
+ "train_moe_sha256": "287bee32718f74d333f48e94c38e5150ed70921f47e15871940c83ce7e13d22a",
10
+ "dataset_revisions": {
11
+ "wikimedia/wikipedia": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa",
12
+ "ZurichNLP/SwissGov-RSD": "f16aa0536811b8c0cf975fede21eec5996e4b3c1",
13
+ "eljuanina/VotingBooklets-v1": "8db11c674d6993aee084fe65ed3ab6d9ee102fd7",
14
+ "eljuanina/VotingBooklets-Diamond-v1": "476c96e74a72b65122cf6fc01943be125f7f3324"
15
+ },
16
  "training_triplets": 31483,
17
  "unique_prompted_texts": 56324,
18
+ "moe_training_max_seq_length": 512,
19
  "inference_max_seq_length": 1024,
20
  "expert_names": [
21
  "de",
 
31
  "language_loss_weight": 0.05,
32
  "balance_loss_weight": 0.01,
33
  "head_learning_rate": 0.0002,
34
+ "head_batch_size": 1024,
35
+ "head_epochs": 5,
36
  "matryoshka_dims": [
37
  1024,
38
  768,
 
48
  "training_history": [
49
  {
50
  "epoch": 1,
51
+ "loss": 2.023314436276754,
52
+ "retrieval_loss": 1.9348004142443338,
53
+ "language_loss": 1.7696655670801797,
54
+ "balance_loss": 0.0030751408770205066,
55
+ "lr": 0.00018090169943749468
56
  },
57
  {
58
  "epoch": 2,
59
+ "loss": 1.9000611066818238,
60
+ "retrieval_loss": 1.8191587686538697,
61
+ "language_loss": 1.616682501633962,
62
+ "balance_loss": 0.006820600215966503,
63
+ "lr": 0.00013090169943749468
64
  },
65
  {
66
  "epoch": 3,
67
+ "loss": 1.8358161449432373,
68
+ "retrieval_loss": 1.7584825873374939,
69
+ "language_loss": 1.5452160716056824,
70
+ "balance_loss": 0.007276078267022967,
71
+ "lr": 6.909830056250522e-05
72
+ },
73
+ {
74
+ "epoch": 4,
75
+ "loss": 1.7946220676104228,
76
+ "retrieval_loss": 1.7193971276283264,
77
+ "language_loss": 1.5032105565071106,
78
+ "balance_loss": 0.00644152524570624,
79
+ "lr": 1.909830056250526e-05
80
+ },
81
+ {
82
+ "epoch": 5,
83
+ "loss": 1.7804000655810037,
84
+ "retrieval_loss": 1.7059372186660766,
85
+ "language_loss": 1.4880693872769675,
86
+ "balance_loss": 0.0059369114848474664,
87
  "lr": 0.0
88
  }
89
  ]