---
language: en
license: cc-by-4.0
library_name: spacy
base_model: en_core_web_md
pipeline_tag: token-classification
tags:
  - spacy
  - ner
  - token-classification
  - legal
  - pii
  - privacy
  - redaction
---

# md_legal_ft3 — legal-text PERSON/ORG NER (false-positive-reduced)

A fine-tune of spaCy **`en_core_web_md`** that **reduces PERSON/ORG false positives on legal documents**
for PII redaction, while staying CPU-fast and drop-in. It targets the two FP classes stock `md`
over-tags on contracts/court text:

- **Section headers** — `RECITALS`, `WITNESSETH`, `GOVERNING LAW` …
- **Defined-term roles** — "the Company", "the Distributor", "the Disclosing Party" …

It is a replacement for `en_core_web_md` **on legal text only**. On non-legal input it regresses
ordinary-org recall by ~40% — fall back to stock `en_core_web_md` there (see **Limitations**).

> **Honesty note.** Every metric below was attacked by an independent adversarial reviewer before
> acceptance; that process caught and killed six inflated claims (listed under *Integrity*). The
> headline contract-precision number rests on **7 hand-adjudicated documents** — a large effect on a
> small sample. **Validate on your own documents before trusting it for redaction.**

---

## TL;DR

| | stock `md` | **`md_legal_ft3`** |
|---|---|---|
| Contract precision (false-tag rate) | 0.37 | **0.82** (FP share 63% → 18%) |
| Header/role FPs (full contract set) | 347 | **23** (~15×) |
| PERSON recall, **unseen** names | 0.857 | **0.920** (generalizes) |
| Court (ECHR) ALL F1 | 0.70 | **0.89** |
| Speed (CPU) | md-speed | **md-speed** |
| ⚠️ General-domain ORG recall (non-legal) | 0.45 | **0.26** (legal-only) |
| ⚠️ Commercial-ORG distinct recall (contracts) | 0.92 | **0.78** |

> The commercial-ORG recall trade is **mostly recoverable** with the optional, non-neural
> `defterm.py` layer shipped alongside (defined-term extraction + corporate-suffix detection +
> alias propagation): strict distinct-recall **0.72 → 0.84** over `ft3`, at no measured precision
> cost and no FP-trap regression. See **Optional: defined-term layer**.

---

## Intended use

- **In scope:** PERSON / ORG NER on **English legal text** (contracts, agreements, court/litigation
  filings), e.g. as the PER/ORG detector in a PII-redaction pipeline where regex/Presidio handle other
  PII types. CPU, ~drop-in for `en_core_web_md`.
- **Out of scope:** general-domain text (news, web, bios) — use stock `en_core_web_md`; non-English;
  entity types other than PERSON/ORG; any use where **missing a third-party company is unacceptable**
  without your own validation (see the recall trade below).

## How to use

```python
import spacy
nlp = spacy.load("md_legal_ft3")          # LEGAL TEXT ONLY
ents = [(e.text, e.label_) for e in nlp(text).ents if e.label_ in ("PERSON", "ORG")]
# For non-legal input, route to stock en_core_web_md instead.
```

## Training data — all public, no private/proprietary data

| source | what | license |
|---|---|---|
| **CUAD** (`chenghao/cuad_qa`) | public SEC-EDGAR commercial contracts; PERSON/ORG labels generated by DeepSeek over the public text | CC-BY-4.0 |
| **TAB-ECHR** (`mattmdjaga/text-anonymization-benchmark-train`) | public European Court of Human Rights judgments, human gold | research/MIT |
| **CoNLL-2003** (`Davlan/conll2003_noMISC`) | public news, used as anti-forgetting rehearsal | research |

**Label policy (harmonized across domains):** tag *specific named* entities; leave *bare generic roles*
(`the Company`, `the Court`, `the Government`) and *section headers* unlabeled. Domains are balanced so
court text can't drown contracts (court capped, contracts oversampled 3×, general PERSON 2×).

No confidential or proprietary documents were used in training, evaluation, or any uploaded artifact.

---

## Evaluation — what it does

Measured stock `md` → `md_legal_ft3`:

- **Contract precision 0.37 → 0.82** (7-doc hand gazetteer); header/role FPs **347 → 23**.
- **Trap set** (adversarial role/header sentences): FP-suppression **53% → 100%**.
- **PERSON recall on unseen names 0.857 → 0.920** — *generalizes* to names never seen in training
  (the transferable win; CI-significant on TAB-ECHR).
- **PERSON distinct-party recall (contracts) 1.00 → 1.00**; **agency recall 1.00 → 1.00** — misses
  zero people and zero agencies (e.g. SEC) in the eval set.
- **Court (ECHR) ALL F1 0.70 → 0.89** (PERSON 0.84→0.93, ORG 0.63→0.85; TAB human gold).
- **Label assignment is cleaner** (ORG→PERSON mislabel 10.7% → 7.6%).
- **Latency:** unchanged from `md` (CNN/tok2vec) — ~6 ms on a short clause, ~200–320 ms on a long
  multi-page judgment, CPU.

## Limitations — what it misses (material; read before shipping)

- **⚠️ Legal-only.** General-domain ORG recall (CoNLL-2003 news) **0.45 → 0.26** (−40% relative). It
  misses ordinary orgs (sports clubs, brands). **Route non-legal text to stock `en_core_web_md`.**
- **⚠️ Commercial-ORG recall trade (model alone).** `ft3` by itself drops some commercial third-party
  orgs (MLB, Marrero Terminal LLC, the Variable Insurance Funds…) — **no persons, no agencies**. *Most
  of this is recovered by the optional `defterm.py` layer* (below); the model alone is the conservative
  floor. A missed party is still a missed party — **validate on your corpus.**
- **ORG does not generalize (in the model).** Unseen-ORG recall in the *model* even on legal text
  **0.66 → 0.61** — ORG competence is per-domain memorization, not a transferable model skill. But most
  legal orgs are recoverable *structurally* (they're defined terms or carry corporate suffixes) — that's
  what `defterm.py` exploits, no retraining. (PERSON and role-suppression *do* transfer in the model.)
- **Residual FPs / precision honesty.** Reported precision (0.82) is **span-level on a 7-doc hand
  gazetteer**; counted **per distinct entity it is ≈0.62**, and the hand adjudicator is itself lenient.
  Treat all absolute precision/recall here as soft n=7 estimates, not pinned numbers.
- **Known FN failure modes:** "The Company Store" — a real org whose name *is* a generic role phrase —
  is the genuinely hard residual. (A bare unseen org like "OpenAI shall provide…" is recoverable via an
  obligation-subject rule; see `actor` in the layer notes, but that rule is lexicon-dependent and not
  enabled by default.)
- **Boundary pathologies** (merged/fragment spans, address-as-ORG) persist from base `md`.

## Optional: defined-term layer (`defterm.py`) — recovers commercial-ORG recall, no retraining

A small **non-neural** layer that runs after the model and recovers most of the org recall `ft3`
trades away — by exploiting document *structure*, not world knowledge:

1. **Defined-term extraction** — `NAME ("ALIAS")` / `NAME, a Delaware LLC ("ALIAS")` → tag NAME +
   identifying ALIAS (acronyms like `MLB`, `FDA`), correctly stripping the `, a <jurisdiction> <type>`
   descriptive tail.
2. **Corporate-suffix generator** — `X LLC / Inc / Corp / Fund …` is an org whether defined or not,
   gated by a role/header/concept filter so `Operating Company`, `Confidential Information Trust`,
   `Reserve Fund` are **not** tagged.
3. **Alias propagation** — every accepted surface tagged at all its occurrences.

**Measured (`ft3` → `ft3+defterm`, 7-doc hand gazetteer):** strict distinct-recall **0.72 → 0.84**
(+0.116, recovers ~11 real orgs: FDA, Cisco, MLB, FIIOC, the Variable Insurance Funds, OMEGA
REFINING/Marrero Terminal…), per-entity precision **0.62 → 0.64** (no cost), trap FP-suppression
**100% → 100%**.

```python
from defterm import defterm_predict     # = ft3 ∪ structural layer
ents = defterm_predict(text)            # [(start, end, label), ...]
```

**Caveats (honest):** this is an **n=7** result; the gain is real and survives strict matching + an
adversarial audit, but it is *modest and small-sample*. The FP-safety of the suffix generator is a
hand-maintained lexicon (`ROLE`/`HEADER`/`CONCEPT`/`SUFFIX_STOP`) — `fp_traps.py` is the regression
guard; extend it when you find a leak. **Do not** enable the blanket capitalized-NP generator (it
craters precision); only the targeted generators above are safe.

## How to extend to a new legal domain

1. LLM-label that domain's docs for PERSON/ORG (verbatim → locate), **same harmonized policy** (tag
   specific names, leave generic roles unlabeled).
2. **Mix with the existing harmonized data** (keep contracts + court so they don't regress).
3. Retrain `md`; eval on held-out docs of that domain **and** the existing evals. ORG competence is
   per-domain; PERSON and role-suppression transfer for free.

## Integrity / methodology

Adversarial review on every claim caught and killed: (1) a circular self-distillation metric (graded on
the same word-list it trained on), (2) an out-of-domain-only "rules beat fine-tuning" verdict, (3) an
ECHR-memorization ORG gain misread as universal, (4) an 18-sentence trap set reporting 93%
FP-suppression for a candidate-generation pipeline whose *real* contract precision was 0.25, (5) a
non-significant n=7 contract-precision gap stated as proven, (6) a latency figure that was multi-arm
contention noise. The numbers above are what survived that process.

## Provenance / licensing

Derived from spaCy `en_core_web_md` (MIT). Training-data attribution: **CUAD (CC-BY-4.0)** requires
attribution; TAB-ECHR and CoNLL-2003 are research datasets. Contract labels are DeepSeek-generated
annotations over public text. This repository is **private**.