README.md

---
library_name: sentence-transformers
pipeline_tag: sentence-similarity
license: apache-2.0
base_model: andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir
datasets:
- wikimedia/wikipedia
- eljuanina/VotingBooklets-v1
- ZurichNLP/SwissGov-RSD
language:
- de
- en
- fr
- it
- rm
- gsw
tags:
- sentence-transformers
- embeddings
- retrieval
- multilingual
- swiss
- apertus
- apertus-1.1
- bidirectional
- matryoshka
- moe
- language-moe
- sparse-routing
---

# Apertus v1.1 Swiss Embed — Language-MoE

A standalone Sentence Transformers retrieval model based on
[`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir).

The model retains the trained **dense bidirectional Apertus encoder backbone** and replaces
the parent's single dense embedding projection with a learned **Language-MoE projection
head**.

> This release is a **dense-backbone + sparse Language-MoE head**, not a backbone-level
> native MoE model.

## Architecture

```text
plain retrieval text
    ↓
bidirectional Apertus encoder
    ↓
mean pooling
    ↓
Language-MoE router
    ↓
Top-2 of:
DE / EN / FR / IT / RM / GSW / shared
    ↓
1024d projection
    ↓
L2 normalization
```

| Property | Value |
|---|---|
| Total parameters | **445,696,559 (0.446B)** |
| Active parameters / sentence | **440,453,679** |
| Language-MoE head parameters | **7,347,207** |
| Active head parameters / sentence | **2,104,327** |
| Router parameters | 7,175 |
| Parameters / expert | 1,048,576 |
| Experts | 7 |
| Active experts / sentence | 2 |
| Native embedding dimension | **1024** |
| Matryoshka dimensions | `[1024, 768, 512, 256]` |
| MoE training max length | **512 tokens** |
| Inference max length | **1024 tokens** |
| Sparse/reference max abs diff | `1.431e-06` |

## Sentence Transformers usage

`LanguageMoE` is a custom Sentence Transformers module, so load the trusted repository with
`trust_remote_code=True`:

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
"andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir-langmoe",
trust_remote_code=True,
)

queries = model.encode_query([
"Welche Aufgaben hat der Bundesrat?"
])

documents = model.encode_document([
"Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
"Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
])

scores = model.similarity(queries, documents)
print(scores)
```

For a compact Matryoshka representation:

```python
embeddings = model.encode_query(
texts,
truncate_dim=512,
)
```

The stored retrieval prefixes are:

- query: `query: `
- document: `passage: `

No chat template is used.

## Lineage

| Property | Value |
|---|---|
| Original source | [`swiss-ai/Apertus-v1.1-0.5B`](https://huggingface.co/swiss-ai/Apertus-v1.1-0.5B) |
| Dense embedding parent | [`andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir`](https://huggingface.co/andreasmartin/apertus-v1.1-swiss-embed-0.4b-bidir) |
| Dense parent revision | `875c1f99d7b20e261ea43b3586fdde5f8d5648e5` |
| Apertus lineage | [`swiss-ai/Apertus-8B-2509`](https://huggingface.co/swiss-ai/Apertus-8B-2509) |

The repository contains the complete transformer/tokenizer/pooling/MoE/normalization pipeline.
The dense parent is **not fetched at inference time**.

## Language-MoE training

The dense embedding backbone was frozen. Its pooled hidden states were cached once and the
router + expert projections were trained for the controlled MoE-head ablation.

See `training_metadata.json` for the exact cache/training history and
`router_diagnostics.json` for held-out routing counts.

Training data:

- [`wikimedia/wikipedia`](https://huggingface.co/datasets/wikimedia/wikipedia)
- [`eljuanina/VotingBooklets-v1`](https://huggingface.co/datasets/eljuanina/VotingBooklets-v1)
- [`ZurichNLP/SwissGov-RSD`](https://huggingface.co/datasets/ZurichNLP/SwissGov-RSD)

`eljuanina/VotingBooklets-Diamond-v1` is held-out evaluation-only and is therefore not listed in the
training dataset metadata.

Actual training set: **31,483 triplets**.

The MoE stage consumes the shared `train_moe.jsonl` generated together with the Dense
`train.jsonl`; triplets and negatives are identical. The released `data_manifest.json` records
the exact Hugging Face dataset commit SHAs, recipe signature and SHA256 hashes of the processed
files.

## Internal retrieval diagnostics

These metrics are computed by reloading the **saved standalone Sentence Transformers model**.
They are internal development diagnostics, not MTEB/MMTEB benchmark claims.

| Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
|---|---:|---:|---:|---:|---:|
| Swiss monolingual | 1024 | 82.08% | 91.88% | 0.8678 | 0.8517 |
| Swiss monolingual | 512 | 80.00% | 91.88% | 0.8593 | 0.8404 |
| Swiss cross-lingual | 1024 | 30.63% | 96.04% | 0.6627 | 0.5635 |
| Swiss cross-lingual | 512 | 30.42% | 96.04% | 0.6602 | 0.5605 |

Monolingual evaluation queries: **480**  
Cross-lingual evaluation queries: **480**

## Router diagnostics

The held-out routing diagnostics are saved in `router_diagnostics.json`.

Recorded shared-expert Top-k selections in the held-out diagnostic:
**0**.

The declared language is used only as weak auxiliary supervision during MoE-head training;
no language ID is required at inference.

## Limitations

- Sparse routing is currently in the embedding projection head; the Apertus backbone remains dense.
- Wikipedia title→paragraph pairs are pseudo-retrieval supervision.
- Parallel passages provide semantic alignment rather than natural search queries.
- External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
- The MoE head was trained on pooled representations up to
  512 tokens. The packaged parent retains
  1024-token inference capability, but quality beyond
  the MoE training context length has not yet been separately validated.

## License and attribution

Apertus is developed by the Swiss AI Initiative. This is an independent retrieval/MoE
adaptation and not an official Swiss AI Initiative embedding release.