Apertus v1.1 Swiss Embed — Language-MoE

A standalone Sentence Transformers retrieval model based on andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir.

The model retains the trained dense bidirectional Apertus encoder backbone and replaces the parent's single dense embedding projection with a learned Language-MoE projection head.

This release is a dense-backbone + sparse Language-MoE head, not a backbone-level native MoE model.

Architecture

plain retrieval text
    ↓
bidirectional Apertus encoder
    ↓
mean pooling
    ↓
Language-MoE router
    ↓
Top-2 of:
DE / EN / FR / IT / RM / GSW / shared
    ↓
1024d projection
    ↓
L2 normalization

Property	Value
Total parameters	445,696,559 (0.446B)
Active parameters / sentence	440,453,679
Language-MoE head parameters	7,347,207
Active head parameters / sentence	2,104,327
Router parameters	7,175
Parameters / expert	1,048,576
Experts	7
Active experts / sentence	2
Native embedding dimension	1024
Matryoshka dimensions	`[1024, 768, 512, 256]`
MoE training max length	512 tokens
Inference max length	1024 tokens
Sparse/reference max abs diff	`1.431e-06`

Sentence Transformers usage

LanguageMoE is a custom Sentence Transformers module, so load the trusted repository with trust_remote_code=True:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
"andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
trust_remote_code=True,
)

queries = model.encode_query([
"Welche Aufgaben hat der Bundesrat?"
])

documents = model.encode_document([
"Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
"Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
])

scores = model.similarity(queries, documents)
print(scores)

For a compact Matryoshka representation:

embeddings = model.encode_query(
texts,
truncate_dim=512,
)

The stored retrieval prefixes are:

query: query:
document: passage:

No chat template is used.

Lineage

Property	Value
Original source	`swiss-ai/Apertus-v1.1-0.5B`
Dense embedding parent	`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`
Dense parent revision	`875c1f99d7b20e261ea43b3586fdde5f8d5648e5`
Apertus lineage	`swiss-ai/Apertus-8B-2509`

The repository contains the complete transformer/tokenizer/pooling/MoE/normalization pipeline. The dense parent is not fetched at inference time.

Language-MoE training

The dense embedding backbone was frozen. Its pooled hidden states were cached once and the router + expert projections were trained for the controlled MoE-head ablation.

See training_metadata.json for the exact cache/training history and router_diagnostics.json for held-out routing counts.

Training data:

eljuanina/VotingBooklets-Diamond-v1 is held-out evaluation-only and is therefore not listed in the training dataset metadata.

Actual training set: 31,483 triplets.

The MoE stage consumes the shared train_moe.jsonl generated together with the Dense train.jsonl; triplets and negatives are identical. The released data_manifest.json records the exact Hugging Face dataset commit SHAs, recipe signature and SHA256 hashes of the processed files.

Internal retrieval diagnostics

These metrics are computed by reloading the saved standalone Sentence Transformers model. They are internal development diagnostics, not MTEB/MMTEB benchmark claims.

Diagnostic	Dim	Accuracy@1	Recall@10	nDCG@10	MRR@10
Swiss monolingual	1024	82.08%	91.88%	0.8678	0.8517
Swiss monolingual	512	80.00%	91.88%	0.8593	0.8404
Swiss cross-lingual	1024	30.63%	96.04%	0.6627	0.5635
Swiss cross-lingual	512	30.42%	96.04%	0.6602	0.5605

Monolingual evaluation queries: 480
Cross-lingual evaluation queries: 480

Router diagnostics

The held-out routing diagnostics are saved in router_diagnostics.json.

Recorded shared-expert Top-k selections in the held-out diagnostic: 0.

The declared language is used only as weak auxiliary supervision during MoE-head training; no language ID is required at inference.

Limitations

Sparse routing is currently in the embedding projection head; the Apertus backbone remains dense.
Wikipedia title→paragraph pairs are pseudo-retrieval supervision.
Parallel passages provide semantic alignment rather than natural search queries.
External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
The MoE head was trained on pooled representations up to 512 tokens. The packaged parent retains 1024-token inference capability, but quality beyond the MoE training context length has not yet been separately validated.

License and attribution

Apertus is developed by the Swiss AI Initiative. This is an independent retrieval/MoE adaptation and not an official Swiss AI Initiative embedding release.