legal-slm-500m-sft

A 528.5M-parameter instruction-tuned legal & financial assistant, fully fine-tuned from DeependraVerma/slm-500m-base — itself trained completely from scratch on a staged-schedule mix of US case law, SEC filings, and educational web text. This is the scaled-up successor to DeependraVerma/legal-slm-125m-sft, fine-tuned on the exact same curated SFT dataset so the two models are directly comparable question-for-question.

• Repo / full build: github.com/DeependraVerma/legal-slm-125M
• Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
• Base model: DeependraVerma/slm-500m-base
• Predecessor (125M): DeependraVerma/legal-slm-125m-sft

Read "Known limitations" below before using this model for anything. It is measurably better than the 125M model on every shared category, but it still hallucinates confidently on closed-book questions (no source text given) — a real, tested limitation, not a hypothetical one. Never use its output as legal, financial, or factual advice.

What this is

Same two-mode design as the 125M SFT model:

1. Open-book (recommended) — give it a contract excerpt or passage along with your question; it extracts/classifies/answers from that given text. This is the mode with real, measured reliability.
2. Closed-book — a general legal/financial question with no source text. Meaningfully better than the 125M model here too, but still the weaker, riskier mode — see Known limitations.

Model description

Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder, 24 layers / 1152 hidden / 18 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 4,608
Vocabulary	16,384 byte-level BPE (same tokenizer as the 125M build, plus role tokens)
Context length	1,024 tokens
Base checkpoint	DeependraVerma/slm-500m-base (this author's own from-scratch pretrain)

Training data

Deliberately reuses the exact same curated SFT dataset as the 125M model — 21,186 pairs: CUAD contract clause extraction (CC BY 4.0), LEDGAR clause classification (CC BY 4.0), and open-book QA/summarization/extraction over case-law/SEC/educational-web passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher. Reusing the same dataset (rather than rebuilding it) is valid because the tokenized data depends only on the tokenizer, which both models share — it isolates the comparison to "does a bigger, better-pretrained base model produce a better SFT result," not "is the SFT data different."

Train / val split	20,127 / 1,059
Method	full fine-tune (not LoRA), 2 epochs
Epoch 1 val_loss → epoch 2 val_loss	0.1840 → 0.1887 (ticked up — a real, measured overfitting signal on this dataset size; do not push epochs further without new data)

Evaluation

Full held-out validation set (1,059 examples), same methodology as the 125M model's card — extraction/classification against ground truth, general QA judged by an independent local Meta-Llama-3.1-70B-Instruct:

Task	125M	500M
CUAD contract clause extraction (Token-F1)	0.711	0.761
CUAD "clause not present" refusal accuracy	90.1%	87.7%
LEDGAR clause classification (exact-match)	74.1%	77.4%
Case law general Q&A (closed-book)	35.6%	42.2%
SEC filings general Q&A (closed-book)	43.1%	46.9%
Educational web general Q&A (closed-book)	18.2%	30.3%

A direct, paired, per-question comparison (both models answering the identical 1,059 questions) found the 500M model wins net in every single category, but not as a clean sweep — genuine regressions exist alongside the improvements (e.g. CUAD: 46 questions fixed, 25 newly wrong that the 125M model had gotten right). One concrete regression, worth citing directly: asked what changed conditions a defendant cited, the 125M model correctly listed the real conditions from the source text; the 500M model produced a vaguer, legal-sounding but non-responsive answer. Bigger and better on net does not mean strictly better on every question.

Known limitations — read before using

• Open-book contract tasks work well and are the trustworthy mode — measurably better than the 125M model across the board.
• Closed-book general Q&A is improved but still fundamentally limited. A 528.5M-parameter model can store at most ~2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — roughly 132MB of total compressible fact storage, shared across everything it knows. Combined with the fact that most specific facts in its training data (a case's dollar figure, a specific statute citation) appeared only once or twice — and memorization research (Carlini et al., "Quantifying Memorization Across Neural Language Models") shows once-seen facts are memorized only ~0.75% of the time versus 40%+ for facts repeated 500+ times — closed-book precision on rare facts is a hard capacity limitation, not something more training on this same recipe fixes.
• Confident fabrication is real and was directly observed, not hypothetical: specific invented clinical/legal details contradicting the real source, and the vague-non-answer regression cited above.

Never use this model's output as legal, financial, or factual advice. Always treat specific claims as unverified until checked against a primary source, especially in closed-book use.

External benchmark evaluation

This project's own eval scripts (above) are one perspective. To check the same claims against outside measurement, both this model and its 125M predecessor were run through lm-evaluation-harness (the same framework behind a widely used public LLM leaderboard) on five benchmark categories:

Benchmark	Result	Baseline	Read
HellaSwag / ARC-Easy / PIQA (general commonsense)	32.4% / 44.5% / 59.6% (acc_norm)	n/a	Normal range for a model this size — not a legal claim, a peer-comparison reference point
MMLU professional_law / jurisprudence / international_law	24.7% / 25.9% / 24.0% (acc)	25% (4-choice)	At the random-chance floor — no real legal knowledge memorized, exactly as the caveats above already say
CaseHOLD (pick the correct legal holding, via LexGLUE)	19.9% (acc_norm)	20% (5-choice)	Right at random chance — real closed-book legal reasoning is not something this model can do, stated as plainly as a number can
LegalBench — 12-task contract NLI/QA subset	53.3% mean acc	50% (binary)	Modest, genuine positive signal — matches this model's actual strength (given text, answer about that text), though individual tasks are mixed, not a clean sweep vs. the 125M model

LegalBench isn't in lm-eval-harness's default task set, so a 12-task, contract-focused subset was hand-configured using the benchmark's own official prompts, and checked for training-data contamination with an n-gram overlap scan against the full pretraining corpus before trusting the result (a naive check first over-flagged common legal boilerplate — standard-of-review language, ToS disclaimer text — as "contamination"; a stricter re-check traced the real overlap to 5 of 1,853 documents, all explainable — e.g. a real, publicly available company's Terms of Service — and none affecting the reported numbers). SCOTUS, ECtHR, and EUR-LEX (also part of LexGLUE) were excluded: those documents run tens of thousands of tokens, far beyond this model's 1,024-token context, and scoring on a truncated sliver of a document isn't a meaningful number.

Full methodology, task configs, and raw logs: github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-500m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-500m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

Citation

@misc{verma2026legalslm500msft,
  author = {Deependra Verma},
  title  = {legal-slm-500m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 528.5M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-500m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face