inlegalbert-sc-ner-silver

Named entity recognition for Indian Supreme Court judgments — v0.1 silver baseline.

Trained on 33,000 Supreme Court judgments (1950–2024). Recognises 13 entity types: statutes, provisions, case citations, judges, petitioners, respondents, lawyers, courts, organisations, geopolitical entities, dates, witnesses, and other persons.

Silver-only baseline. Annotations are automatically generated (regex + transformer NER + gazetteer enrichment), not human-verified. Gold-annotated v1.0 is in progress.

Usage

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="evolawyer/inlegalbert-sc-ner-silver",
    aggregation_strategy="simple",
)

text = (
    "The appellant M/S Emaar MGF Land Ltd. challenged the order under s.21 "
    "of the Consumer Protection Act, 1986, relying on (2017) 15 SCC 720."
)

for e in ner(text):
    print(e["entity_group"], "|", e["word"], "|", round(e["score"], 3))

Labels

STATUTE · PROVISION · CASE_CITATION · JUDGE · PETITIONER · RESPONDENT · LAWYER · COURT · ORG · GPE · DATE · OTHER_PERSON · WITNESS

IOB2 tag set: 27 tags total (O + B-/I- × 13).

Training

Property	Value
Base model	law-ai/InLegalBERT (BERT-base, 110M parameters)
Head	Linear softmax (AutoModelForTokenClassification)
Training data	~34,700 silver-annotated chunks from 33k judgments
Epochs	3
Max length	512 tokens
Stride (train)	128 (overlapping chunks)
Stride (val)	512 (non-overlapping)
Batch size	8 (fp16 + gradient checkpointing)
Learning rate	2e-5
Hardware	Kaggle T4

Evaluation

Evaluated on a non-overlapping held-out validation split (stride=512, ~500 documents). F1 is slightly conservative: entities that land exactly on a 512-token chunk boundary are scored as FP+FN, affecting <1% of entities.

Entity	Precision	Recall	F1	Support
CASE_CITATION	96.72%	98.82%	97.76%	3,821
PROVISION	96.09%	96.60%	96.35%	20,248
STATUTE	90.08%	93.88%	91.94%	8,187
LAWYER	73.60%	75.77%	74.67%	3,982
JUDGE	66.80%	69.36%	68.06%	1,978
DATE	51.98%	58.74%	55.15%	3,289
RESPONDENT	51.28%	49.62%	50.44%	1,731
COURT	46.91%	54.31%	50.34%	1,033
WITNESS	44.93%	55.77%	49.77%	762
OTHER_PERSON	43.20%	51.78%	47.11%	4,266
PETITIONER	55.07%	37.64%	44.71%	1,573
ORG	42.36%	40.37%	41.34%	2,128
GPE	38.07%	35.17%	36.56% ⚠	1,197
micro avg	77.54%	79.84%	78.67%	54,195
macro avg	61.31%	62.91%	61.86%	—
weighted avg	78.00%	79.84%	78.80%	—

Comparison baseline: OpenNyAI's 2022 model (RoBERTa + transition-based parser, gold-annotated) achieved 91.1% overall strict F1. This v0.1 silver baseline is not directly comparable — different test sets, different annotation quality, different corpus scope. CASE_CITATION at 97.76% already exceeds OpenNyAI's PRECEDENT score of 80.1% by +17 points. Official head-to-head comparison runs in v1.0 on the locked 25-file test set.

Data Sources

Silver labels produced by four automatic annotation pipelines merged per document:

• Metadata + header projection — case metadata JSONs mapped to character offsets via RapidFuzz; produces JUDGE, PETITIONER, RESPONDENT
• Regex — 14-pattern case citation extractor + statute/provision extractor; produces CASE_CITATION, STATUTE, PROVISION
• Transformer NER — OpenNyAI en_legal_ner_trf (InLegalBERT-based), offset-corrected; produces LAWYER, COURT, ORG, GPE, DATE, OTHER_PERSON, WITNESS
• Gazetteer — 858 Central Acts with alias resolution; adds and confirms STATUTE spans

Training data: evolawyer/indian-sc-judgments-ner-silver. The 25 locked test files are excluded.

Known Limitations

• GPE (36.56% F1) and ORG (41.34% F1) — role overlap. In Indian legal text, entities like "State of Maharashtra" or "Union of India" constantly appear as GPE, PETITIONER, RESPONDENT, or ORG depending on context. The linear token classification head cannot resolve these overlapping roles. Primary focus of v1.0 CRF head.
• Positional bias. Silver training data has repetitive header structures. Performance degrades when parties appear mid-document.
• Token fragmentation. Rare or hyphenated names can be split across subwords without a CRF sequence layer.
• Silver annotations only. Labels are automatically generated. Entity boundaries may drift at span edges.
• Pre-1990 OCR noise. Judgments from 1950–1989 vary in OCR quality.
• 25-file test set is locked. The 78.67% figure is a validation estimate, not the final benchmark score.

License

Apache 2.0. Base model law-ai/InLegalBERT is MIT licensed (compatible with Apache 2.0).

---

evolawyer.com · GitHub