You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

md_legal_ft3 — legal-text PERSON/ORG NER (false-positive-reduced)

A fine-tune of spaCy en_core_web_md that reduces PERSON/ORG false positives on legal documents for PII redaction, while staying CPU-fast and drop-in. It targets the two FP classes stock md over-tags on contracts/court text:

Section headers — RECITALS, WITNESSETH, GOVERNING LAW …
Defined-term roles — "the Company", "the Distributor", "the Disclosing Party" …

It is a replacement for en_core_web_md on legal text only. On non-legal input it regresses ordinary-org recall by ~40% — fall back to stock en_core_web_md there (see Limitations).

Honesty note. Every metric below was attacked by an independent adversarial reviewer before acceptance; that process caught and killed six inflated claims (listed under Integrity). The headline contract-precision number rests on 7 hand-adjudicated documents — a large effect on a small sample. Validate on your own documents before trusting it for redaction.

TL;DR

	stock `md`	`md_legal_ft3`
Contract precision (false-tag rate)	0.37	0.82 (FP share 63% → 18%)
Header/role FPs (full contract set)	347	23 (~15×)
PERSON recall, unseen names	0.857	0.920 (generalizes)
Court (ECHR) ALL F1	0.70	0.89
Speed (CPU)	md-speed	md-speed
⚠️ General-domain ORG recall (non-legal)	0.45	0.26 (legal-only)
⚠️ Commercial-ORG distinct recall (contracts)	0.92	0.78

The commercial-ORG recall trade is mostly recoverable with the optional, non-neural defterm.py layer shipped alongside (defined-term extraction + corporate-suffix detection + alias propagation): strict distinct-recall 0.72 → 0.84 over ft3, at no measured precision cost and no FP-trap regression. See Optional: defined-term layer.

Intended use

In scope: PERSON / ORG NER on English legal text (contracts, agreements, court/litigation filings), e.g. as the PER/ORG detector in a PII-redaction pipeline where regex/Presidio handle other PII types. CPU, ~drop-in for en_core_web_md.
Out of scope: general-domain text (news, web, bios) — use stock en_core_web_md; non-English; entity types other than PERSON/ORG; any use where missing a third-party company is unacceptable without your own validation (see the recall trade below).

How to use

import spacy
nlp = spacy.load("md_legal_ft3")          # LEGAL TEXT ONLY
ents = [(e.text, e.label_) for e in nlp(text).ents if e.label_ in ("PERSON", "ORG")]
# For non-legal input, route to stock en_core_web_md instead.

Training data — all public, no private/proprietary data

source	what	license
CUAD (`chenghao/cuad_qa`)	public SEC-EDGAR commercial contracts; PERSON/ORG labels generated by DeepSeek over the public text	CC-BY-4.0
TAB-ECHR (`mattmdjaga/text-anonymization-benchmark-train`)	public European Court of Human Rights judgments, human gold	research/MIT
CoNLL-2003 (`Davlan/conll2003_noMISC`)	public news, used as anti-forgetting rehearsal	research

Label policy (harmonized across domains): tag specific named entities; leave bare generic roles (the Company, the Court, the Government) and section headers unlabeled. Domains are balanced so court text can't drown contracts (court capped, contracts oversampled 3×, general PERSON 2×).

No confidential or proprietary documents were used in training, evaluation, or any uploaded artifact.

Evaluation — what it does

Measured stock md → md_legal_ft3:

Contract precision 0.37 → 0.82 (7-doc hand gazetteer); header/role FPs 347 → 23.
Trap set (adversarial role/header sentences): FP-suppression 53% → 100%.
PERSON recall on unseen names 0.857 → 0.920 — generalizes to names never seen in training (the transferable win; CI-significant on TAB-ECHR).
PERSON distinct-party recall (contracts) 1.00 → 1.00; agency recall 1.00 → 1.00 — misses zero people and zero agencies (e.g. SEC) in the eval set.
Court (ECHR) ALL F1 0.70 → 0.89 (PERSON 0.84→0.93, ORG 0.63→0.85; TAB human gold).
Label assignment is cleaner (ORG→PERSON mislabel 10.7% → 7.6%).
Latency: unchanged from md (CNN/tok2vec) — ~6 ms on a short clause, ~200–320 ms on a long multi-page judgment, CPU.

Limitations — what it misses (material; read before shipping)

⚠️ Legal-only. General-domain ORG recall (CoNLL-2003 news) 0.45 → 0.26 (−40% relative). It misses ordinary orgs (sports clubs, brands). Route non-legal text to stock en_core_web_md.
⚠️ Commercial-ORG recall trade (model alone). ft3 by itself drops some commercial third-party orgs (MLB, Marrero Terminal LLC, the Variable Insurance Funds…) — no persons, no agencies. Most of this is recovered by the optional defterm.py layer (below); the model alone is the conservative floor. A missed party is still a missed party — validate on your corpus.
ORG does not generalize (in the model). Unseen-ORG recall in the model even on legal text 0.66 → 0.61 — ORG competence is per-domain memorization, not a transferable model skill. But most legal orgs are recoverable structurally (they're defined terms or carry corporate suffixes) — that's what defterm.py exploits, no retraining. (PERSON and role-suppression do transfer in the model.)
Residual FPs / precision honesty. Reported precision (0.82) is span-level on a 7-doc hand gazetteer; counted per distinct entity it is ≈0.62, and the hand adjudicator is itself lenient. Treat all absolute precision/recall here as soft n=7 estimates, not pinned numbers.
Known FN failure modes: "The Company Store" — a real org whose name is a generic role phrase — is the genuinely hard residual. (A bare unseen org like "OpenAI shall provide…" is recoverable via an obligation-subject rule; see actor in the layer notes, but that rule is lexicon-dependent and not enabled by default.)
Boundary pathologies (merged/fragment spans, address-as-ORG) persist from base md.

Optional: defined-term layer (`defterm.py`) — recovers commercial-ORG recall, no retraining

A small non-neural layer that runs after the model and recovers most of the org recall ft3 trades away — by exploiting document structure, not world knowledge:

Defined-term extraction — NAME ("ALIAS") / NAME, a Delaware LLC ("ALIAS") → tag NAME + identifying ALIAS (acronyms like MLB, FDA), correctly stripping the , a <jurisdiction> <type> descriptive tail.
Corporate-suffix generator — X LLC / Inc / Corp / Fund … is an org whether defined or not, gated by a role/header/concept filter so Operating Company, Confidential Information Trust, Reserve Fund are not tagged.
Alias propagation — every accepted surface tagged at all its occurrences.

Measured (ft3 → ft3+defterm, 7-doc hand gazetteer): strict distinct-recall 0.72 → 0.84 (+0.116, recovers ~11 real orgs: FDA, Cisco, MLB, FIIOC, the Variable Insurance Funds, OMEGA REFINING/Marrero Terminal…), per-entity precision 0.62 → 0.64 (no cost), trap FP-suppression 100% → 100%.

from defterm import defterm_predict     # = ft3 ∪ structural layer
ents = defterm_predict(text)            # [(start, end, label), ...]

Caveats (honest): this is an n=7 result; the gain is real and survives strict matching + an adversarial audit, but it is modest and small-sample. The FP-safety of the suffix generator is a hand-maintained lexicon (ROLE/HEADER/CONCEPT/SUFFIX_STOP) — fp_traps.py is the regression guard; extend it when you find a leak. Do not enable the blanket capitalized-NP generator (it craters precision); only the targeted generators above are safe.

How to extend to a new legal domain

LLM-label that domain's docs for PERSON/ORG (verbatim → locate), same harmonized policy (tag specific names, leave generic roles unlabeled).
Mix with the existing harmonized data (keep contracts + court so they don't regress).
Retrain md; eval on held-out docs of that domain and the existing evals. ORG competence is per-domain; PERSON and role-suppression transfer for free.

Integrity / methodology

Adversarial review on every claim caught and killed: (1) a circular self-distillation metric (graded on the same word-list it trained on), (2) an out-of-domain-only "rules beat fine-tuning" verdict, (3) an ECHR-memorization ORG gain misread as universal, (4) an 18-sentence trap set reporting 93% FP-suppression for a candidate-generation pipeline whose real contract precision was 0.25, (5) a non-significant n=7 contract-precision gap stated as proven, (6) a latency figure that was multi-arm contention noise. The numbers above are what survived that process.

Provenance / licensing

Derived from spaCy en_core_web_md (MIT). Training-data attribution: CUAD (CC-BY-4.0) requires attribution; TAB-ECHR and CoNLL-2003 are research datasets. Contract labels are DeepSeek-generated annotations over public text. This repository is private.

Downloads last month: -