teex-pt/AMALIA-9B-0626-DPO-LoRA-legal-v2

AMALIA-9B LoRA — legal-v2 (grounded legal citation pilot, scaled)

LoRA adapter from a research pilot on AMALIA-9B-0626-DPO, targeting grounded legal citation: answer only from provided excerpts, cite the exact article with [F#] tags, and refuse explicitly when the excerpts don't contain the answer. Same recipe as legal-v1, ~2x the training data. Full methodology and every pilot's report: github.com/teex-pt/pt-amalia.

Status: research pilot, closer to a clean accept than legal-v1. Target metric climbed further; only two low-power secondary categories remain outside this project's strict acceptance tolerance, both already diagnosed as generic noise rather than a legal-v2-specific effect.

Why RAG-first, not closed-book legal QA

This project's own exam-domain pilots (iave-v1/iave-v2) showed closed-book fact-recall SFT on small specialized data has a low ceiling — answer format moved, precision didn't. So this adapter trains a different contract entirely: answer only from context provided at inference time, cite it, or say explicitly that the context doesn't answer the question — the same shape a retrieval-augmented legal service already needs, not a model that memorized the law.

Results (vs. baseline, merge-75, and legal-v1)

merge-75 is this project's general-purpose champion (no legal or exam training in its lineage — included to separate "any adapter helps a little" from a domain-specific effect).

Category	Baseline	merge-75	legal-v1	legal-v2
legal citation (target, n=50)	16.0%	16.0%	82.0%	94.0%
legal refusal (n=10)	100%	90.0%	100%	100%
arithmetic (n=100)	46.0%	51.0%	50.0%	58.0%
format (n=30)	73.3%	80.0%	80.0%	80.0%
honesty (n=100)	50.0%	96.0%	78.0%	95.0%
variety (n=30)	86.7%	93.3%	83.3%	86.7%
honesty control (n=36)	100%	100%	97.2%	97.2%
mcq / exam (n=37)	29.7%	27.0%	27.0%	27.0%

Target kept climbing with more data (+12pp over legal-v1, +78pp over baseline). variety fully recovered to baseline (83.3%→86.7%) simply by using the full available general-anchor pool (751 rows vs. legal-v1's 400) — turned out to be under-sampling, not a need for smarter category stratification. honesty_control and mcq land at exactly the same rate as legal-v1 (0.0pp difference both ways), reinforcing they're low-power noise (1 flipped item out of 36/37 each) rather than something more legal data should move — mcq in particular shows the identical drop on merge-75 too, despite zero legal/exam exposure there.

honesty originally read as a regression vs. legal-v1 (78%→68%). It wasn't: legal-v2 simply shifted its default fictional-entity refusal phrasing from "Não conheço X" to "Não identifico X" (both genuine, calibrated refusals), and the checker's keyword list didn't recognize the second one. Manual read of all 16 flipped items confirmed every one was a real refusal, not a new confabulation; fixed by adding "não identifico" to harness/verifiers.py's marker list and re-scoring the stored responses (no re-inference needed — scoring is a pure function of response text). Corrected: 78.0%→95.0% (+17pp), in line with every other category in this pilot rather than the one outlier it first looked like. Full diagnosis: JOURNAL.md, 2026-07-14.

Comparison with other general-purpose models

Zero-shot (no legal-specific training, same fair test used for AMALIA's own untrained baseline above) on the legal_cita/legal_refusal benchmark — none of these models have seen the [F#] citation convention, from open weights up to a frontier commercial model.

Metric	AMALIA-9B (no adapter)	Ministral-3-14B-Reasoning	Mistral-Small-3.2-24B	Claude Sonnet 5 (zero-shot)	Claude Sonnet 5 (few-shot)	AMALIA-9B + legal-v2
legal citation (target, n=50)	16.0%	22.0%	4.0%	18.0%	22.0%	94.0%
legal refusal (n=10)	100%	100%	100%	100%	90.0%	100%

Neither model size nor overall capability predicts this behavior: a frontier model (Sonnet 5) lands at 18.0%, in the same range as AMALIA's own untrained baseline (16.0%) and clearly below Ministral-3-14B (22.0%), while the larger Mistral-Small-24B does considerably worse still (4.0%). Reading Sonnet 5's actual failures confirms the pattern: it correctly extracts and explains the right content in clear, well-structured pt-PT prose, it just doesn't spontaneously invent the [F#] tag syntax — a project-specific convention, not a general capability gap. It's also notably more conservative than the other baselines: 10 of its 41 failures were outright refusals on questions the excerpts actually did support (vs. 3/42 for AMALIA's own baseline), suggesting a stronger default bias toward declining under any uncertainty.

Does showing the format fix it? Not really — the failure mode just changes shape. To test whether the gap is "nobody told it the convention" rather than something fine-tuning was actually needed for, Sonnet 5 was re-run with the real production system prompt used by lexbase.pt (this project's retrieval service), plus one worked example (question, excerpts, and a correctly-cited answer) — pure in-context instruction, zero fine-tuning. (The public harness/run_harness_anthropic.py script ships with an independently-worded equivalent, same rules, not the literal production wording — this specific run used the real one, with explicit authorization, since the point was testing the actual production instruction rather than a generic stand-in.) The aggregate number barely moves (18.0%→22.0%), but why it barely moves is the interesting part: reading the actual failures shows the [F#] tag syntax gets picked up almost immediately (31→3 failures for "never used a tag"), but the model becomes dramatically more conservative in exchange — refusals on genuinely grounded questions jump from 10 to 36 of 50. Fixing the format problem essentially traded it for a much bigger calibration problem: knowing when to cite versus decline isn't something one demonstration teaches, but it's exactly what a small set of diverse training examples (positive and negative) can. All models hold legal_refusal cleanly (100%) except Sonnet 5 few-shot (90.0%, and even that one flip looks like a token-budget truncation mid-explanation rather than a genuine over-citation, on inspection) — so the gap is specifically about citation format and calibration, not grounding discipline in general — and none of it approaches what a small, targeted LoRA pilot achieves (94.0%). Model scale, general capability, and even direct instruction don't substitute for task-specific training on this benchmark.

Tested against real retrieval (lexbase.pt)

Every result above uses a fixed, pre-selected excerpt set. To test something closer to an actual deployment, this adapter was connected to lexbase.pt — the production Portuguese legal search service this project's retrieval index runs on — via its MCP API, on 10 genuinely new, naturally-phrased questions never seen in any training or eval data (e.g. "Uma empresa pode despedir uma trabalhadora grávida?" rather than the benchmark's amendment-summary-style phrasing).

Citation tag usage on real queries: legal-v2 8-9/10, vs. baseline 1/10, Ministral 4/10, Mistral-Small 1/10, Sonnet 5 7/10.

The standout result: on a deliberately off-topic query (a pastry recipe), baseline and Ministral noticed the retrieved excerpts didn't cover it — then answered from general knowledge anyway. legal-v2 and Sonnet 5 both refused correctly; legal-v2's refusal used wording it was never trained on ("consulte uma fonte dedicada à gastronomia portuguesa" vs. the trained template's "consulte diretamente as fontes indicadas") — evidence of a generalized pattern, not a memorized string.

A genuine limitation, found and reported rather than hidden: a manual correctness review (checking answers against what the retrieved excerpts actually say, since no gold answers exist for natural queries) found one real query where lexbase.pt's retrieval missed the actually authoritative article. legal-v2's strict grounding discipline — the same trait that makes it refuse correctly on off-topic questions — meant it stayed faithful to the incomplete shown context instead of filling the gap from background knowledge, while some other models answered more completely by doing exactly that. Root-caused down to how lexbase.pt chunks long enumerated-list articles (one 13-ground dismissal article diluted into a single chunk), with a reproducible test case handed back to the retrieval side.

Full methodology, the complete external-model comparison, and the retrieval diagnosis: eval/results/RAG-INTEGRATION-TEST.md in github.com/teex-pt/pt-amalia.

Example responses

Real outputs from the held-out evaluation set (legal_cita_prompts.jsonl), not cherry-picked beyond "correctly passed and reasonably short to quote."

Grounded citation, multiple excerpts:

PERGUNTA: Revisão e fixação dos valores das taxas do imposto sobre os produtos petrolíferos e energéticos

EXCERTOS: [F1] Revisão e fixação dos valores das taxas do imposto sobre os produtos petrolíferos e energéticos, Artigo 4.º Suspensão de efeitos da Portaria n.º 301-A/2018, de 23 de novembro — «É suspensa a produção de efeitos da Portaria n.º 301-A/2018, de 23 de novembro, enquanto se mantiverem os efeitos da presente portaria.» [F2] ...Artigo 5.º Revogação da Portaria n.º 208-A/2021, de 15 de outubro, e da Portaria n.º 63-A/2022, de 31 de janeiro — «São revogadas a Portaria n.º 208-A/2021, de 15 de outubro, e a Portaria n.º 63-A/2022, de 31 de janeiro.»

RESPOSTA: A alteração introduzida por Lei n.º 10-A/2023, de 2023-05-23, incide sobre o(s) seguinte(s) ponto(s) do(a) Revisão e fixação dos valores das taxas do imposto sobre os produtos petrolíferos e energéticos: [F1] ...Artigo 4.º...: «É suspensa a produção de efeitos da Portaria n.º 301-A/2018, de 23 de novembro, enquanto se mantiverem os efeitos da presente portaria.» [F2] ...Artigo 5.º...: «São revogadas a Portaria n.º 208-A/2021, de 15 de outubro, e a Portaria n.º 63-A/2022, de 31 de janeiro.»

Correct refusal on an unrelated context (question about the 2022 State Budget, excerpts are about asylum/refugee law — deliberately mismatched to test grounding discipline):

PERGUNTA: Orçamento do Estado para 2022

EXCERTOS: [F1] Condições e procedimentos de concessão de asilo ou protecção subsidiária..., Artigo 54.º Direito ao trabalho — «Aos requerentes de asilo ou de proteção subsidiária é assegurado o acesso ao mercado de trabalho...» [F2] ...Artigo 47.º Proibição de expulsar ou repelir — «Ninguém será devolvido, afastado extraditado ou expulso para um país onde seja submetido a torturas ou a tratamentos cruéis ou degradantes.» [F3] ...Artigo 82.º Forma de notificação — (...)

RESPOSTA: Não consigo fundamentar uma resposta nos excertos encontrados. Consulte diretamente as fontes indicadas.

How it was made

1. Data: ~2x legal-v1's mix — 700 grounded-citation examples, 100 deliberately-ungrounded refusal examples, 400 legal-text summarization examples, and the full available pool of general-purpose anchors (arithmetic, format, honesty, variety), roughly a 60/40 target/anchor ratio.
2. Training: mlx_lm.lora, base BF16, 1,000 iterations (~2x legal-v1's, proportional to the larger mix), batch size 1, 16 layers, LoRA rank 8, learning rate 1e-5, max_seq_length=4096 with gradient checkpointing. Peak memory landed at the same ~27GB as legal-v1 — this recipe's memory footprint depends on sequence length/batch size, not dataset size.
3. Evaluation: exact-match deterministic checkers (citation-tag validity against the excerpts actually shown, refusal-phrase matching) — the same code-verifier philosophy as every other pilot in this series, no LLM judging anywhere.

Usage

pip install mlx-lm
mlx_lm.generate --model amalia-llm/AMALIA-9B-0626-DPO --adapter-path <this-repo> \
    --prompt "PERGUNTA: <a sua pergunta>

EXCERTOS:
[F1] <fonte> — <texto do excerto>"

Expects excerpts to be provided in the prompt (retrieval is not part of this adapter) — designed to sit behind a retrieval step, not replace one.

Attribution

Base model by the AMALIA team (Apache 2.0). Adapter, data, and method: teex-pt.