---
license: other
license_name: dilr-internal
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: peft
pipeline_tag: text-generation
tags:
  - clinical
  - information-extraction
  - structured-output
  - json-extraction
  - qlora
  - lora
  - on-prem
language:
  - en
  - es
  - pt
---

# Mira-3 v11 — Clinical Structured Extraction (Qwen3-4B, QLoRA)

**Mira-3** reads a clinical document (lab report, discharge summary, intake form, progress note, …) and returns a single **schema-valid, source-grounded JSON** object. It is a LoRA adapter over `Qwen/Qwen3-4B-Instruct-2507`, built to run **on-prem / offline** on a customer's own GPU — privacy is the point.

This repo is the **v0** artifact: trained on **synthetic + permissively-licensed public data only** (no PHI, no DUA/research-only data). It proves the pipeline; v1 will retrain on a design partner's real documents.

> ⚠️ **Read the honest eval below before using.** v0 is excellent on validity and grounding but has a real-world **recall blind spot on narrative diagnoses/procedures**. See `MIRA3_LEARNINGS.md` for the full analysis and how to fix it.

## Files
- `adapter/` — the shipping adapter (**epoch 3**, best by eval)
- `epochs/epoch_{1,2,3}/` — per-epoch adapters (for reproducibility / best-epoch verification)
- `training/metrics.json`, `training/env_versions.json` — training run + exact dependency stack
- `eval/probe_scorecard_v11.json` — real-doc probe validity + hallucination + leak
- `eval/phase25_gate_v11.json` — Phase-2.5 exit gate verdict (**PASS**)
- `eval/realdoc_eval_v11.json` — the honest labeled real-doc field-F1 (with adjudicated gold)
- `MIRA3_LEARNINGS.md` — detailed, reusable playbook (generalizes to other extraction/OCR verticals, e.g. résumé parsing)

## Training
| | |
|---|---|
| Base | `Qwen/Qwen3-4B-Instruct-2507` (Unsloth 4-bit) |
| Method | QLoRA (4-bit + LoRA r16/α32, dropout 0.05), Unsloth, sequence packing |
| Data | 21k rows: 15k ladder + 3k schema-variant + 3k PII-abstention (synthetic + public) |
| Schedule | 3 epochs, lr 2e-4 cosine, warmup 3%, effective batch 16, max_seq 3072 |
| Compute | Kaggle T4, 2 sessions (~20.7h) via lossless checkpoint/resume |
| Loss | 1.363 → 0.133 (clean cosine decay; natural completion at 867/867 steps) |

## Evaluation (honest)

**Phase-2.5 gate: PASS.** v11 beats the prior production model (Mira-Q2) on all real-doc probes for both JSON validity (100%) and hallucination.

| Probe (real docs) | JSON validity | Hallucination (↓ better) |
|---|---|---|
| mtsamples_282 | 1.00 | 0.28 |
| extraction_relevant_150 | 1.00 | 0.52 |
| synthetic_v2_150 | 1.00 | 0.13 |
| identifier leak | — | **0** everywhere |

**Labeled real-doc field-F1 (the honest generalization metric): 0.61** — on 24 diverse mtsamples docs with double-annotated + adjudicated gold. (The synthetic in-distribution test set scores 1.00 and is **not** a generalization signal — see learnings §4c.)

Per-field: **high precision (0.89–0.98) everywhere, but uneven recall.** Strong on allergies (F1 0.99), encounter date/dept (0.96/0.88), vitals (0.84), medications (P 0.98). **Weak on diagnoses (recall 0.10) and procedures (recall 0.33).**

## Limitations
- **Diagnoses/procedures recall on narrative text is poor.** The model extracts these well from structured `Diagnoses:`-style blocks (as in the synthetic training data) but misses them when stated in prose (real clinical notes). This is a **data-coverage** issue, targeted for v1. Do not rely on v0 for complete diagnosis capture from free-text notes.
- Every output is a **draft for a human** and must be routed through the verifier (schema + grounding + identifier-leak) before use. Never an autonomous decision.
- v0 is trained on synthetic/public data; real-world distribution shift is expected until the v1 retrain.

## Usage
```python
import torch
from unsloth import FastLanguageModel  # load with Unsloth, NOT vanilla PeftModel (4-bit quant mismatch)

model, tok = FastLanguageModel.from_pretrained(
    "dilr/mira-3-v11/adapter", max_seq_length=4096, dtype=torch.float16, load_in_4bit=True)
FastLanguageModel.for_inference(model)

SYSTEM = ("You are a clinical information extraction system. Read the clinical document and "
          "output a single JSON object matching the schema. Extract ONLY information explicitly "
          "stated ... Output valid JSON only - no prose, no markdown.")  # full prompt in the training data

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": document_text}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to("cuda"), max_new_tokens=2048, do_sample=False)
raw = tok.decode(out[0], skip_special_tokens=True)
# NOTE: Qwen3 prepends a <think>...</think> block — strip it before json.loads():
import re, json
pred = json.loads(re.sub(r"^\s*<think>.*?</think>\s*", "", raw.split("assistant")[-1], flags=re.DOTALL))
```

## License & data
LoRA adapter over Qwen3-4B (base: Apache-2.0). Internal dilr.ai artifact; not for redistribution. Trained on synthetic + permissively-licensed public data; **contains no PHI and no patient identifiers**. Eval derived from the public MTSamples corpus.