---
license: apache-2.0
base_model: FacebookAI/xlm-roberta-large
language:
- en
pipeline_tag: token-classification
tags:
- pii
- de-identification
- privacy
- gdpr
- legal
- re-identification
- kp
- europriv-bench
---

# kp-deid-xlmr-560m-legal — a re-identification *protector* for legal text

A 560M XLM-RoBERTa-large token classifier (LoRA-finetuned, adapter merged) for **PII
de-identification**, trained for the EuroPriv-Bench program. It is published not because it tops
detection-F1 — it does not — but because, on the metric that measures privacy harm, it is the best
protector we have found on real legal text.

**Headline (real, external gold):** on the **Text Anonymization Benchmark** (TAB; Pilán et al. 2022 —
real ECHR court judgments, manually annotated), this model leaves **9.5% of DIRECT identifiers
un-redacted** vs the next-best public system's **49.6%** (spaCy) / **50.0%** (Presidio) — i.e. it
leaks **~5× fewer** of the identifiers that directly re-identify a person.

- **Robust:** 3 training seeds → DIRECT-leak {9.5%, 9.9%, 9.9%}.
- **Significant:** paired bootstrap over 264 shared DIRECT subjects → Δ vs runner-up = **−40 points,
  95% CI [−47.7, −32.2]** (fully below 0).
- **Not over-redaction:** it marks 9.1% of tokens as PII (gold = 11.3% — it *under*-tags overall),
  with 4.3% false-positive redaction. It is aimed, not blunt.

## Why detection-F1 is the wrong number, and re-id is the right one

This model sits **mid-table on strict detection-F1 (~0.34)**. That is by design and is the whole
point: strict F1 punishes disagreeing with the gold on a span's exact boundary or entity type — but a
person's privacy does not care about the label, only about whether the identifying string was
redacted. This model reliably *touches* the identifier tokens (removing them) even when it gets the
boundary or type slightly wrong. **If you choose a de-identifier by detection-F1, you can choose one
that leaves half the identifiers behind.** Choose by leak-rate.

## Training data

Trained on **real CJEU judgment *structure*** (from `davidwickerhf/cjeu-opendata`, Apache-2.0) — real
legal document layouts — with **synthetic, checksum-valid PII** spliced in (gold by construction).
This is real legal *structure* with *synthetic* identifiers; it contains **no real personal data**.
CJEU (Court of Justice of the EU) is a different court from ECHR, so the TAB result above is genuine
**zero-shot** generalisation (contamination checked: 0 document overlap).

## Taxonomy

18 GDPR-aligned KP entity types (BIOES): PERSON, NATIONAL_ID, ACCOUNT_ID, ADDRESS, EMAIL, PHONE,
DATE, URL, ORG_PARTY, CASE_NUMBER, COURT, STATUTE_REF, COMPANY_ID, FACILITY, PROVIDER, MRN,
HEALTH_CONDITION, SECRET.

## Intended use & limitations

- **Use:** de-identification of **English (legal) text** where minimising re-identification risk is
  the priority — detect spans, then redact/replace.
- **Scope of the claim:** validated on **one board** (TAB, English legal). Breadth across languages
  and domains is in progress; treat use outside English legal as unvalidated.
- **Research checkpoint** (seed 0 of 3). Not clinical-validated. Strict-F1 is modest — expect
  boundary/type imperfection that does not affect redaction coverage.
- The re-id leak-rate metric is grounded in TAB's own externally-annotated DIRECT/QUASI labels (it is
  not a self-defined construct), but competitors report detection-F1, so head-to-head re-id numbers
  come from scoring all systems through EuroPriv-Bench.

## Links

- Benchmark & leaderboard: [research.klusai.com/leaderboard](https://research.klusai.com/leaderboard)
- Write-up: *Every detector leaked half the IDs on real court text — except one*
  (research.klusai.com/blog)
- Comparand: [klusai/kp-deid-mdeberta-280m](https://huggingface.co/klusai/kp-deid-mdeberta-280m)

*config_status: dev (synthetic training data) / evaluated on real-external-gold (TAB). Not a
peer-reviewed claim yet — see the scorecard for the full method and caveats.*