soybelli/gemma-4-E4B-it-cuad-lora

Gemma 4 E4B — CUAD Contract Review (LoRA)

A LoRA adapter that fine-tunes Gemma 4 E4B (instruct) for legal contract clause review on the CUAD (Contract Understanding Atticus Dataset).

Given a contract excerpt and a clause question (e.g. "Highlight the parts related to 'Governing Law'…"), the model extracts the exact relevant span from the excerpt, or answers "Not found" when the clause is absent.

• Base model: unsloth/gemma-4-E4B-it (trained on the unsloth/gemma-4-E4B-it-unsloth-bnb-4bit 4-bit variant)
• Method: QLoRA SFT with Unsloth + TRL (r=16, alpha=16), language layers only
• Task: extractive QA / clause highlighting over 41 CUAD clause categories
• Adapter size: ~37M trainable params (0.46% of the base)

Results

Evaluated on held-out contracts (10% of CUAD held out by contract, so no context leakage), SQuAD-2.0-style scoring on 80 questions (40 answerable / 40 not-found):

Metric	Base Gemma 4	+ this LoRA	Δ
Overall Exact Match	56.2	76.2	+20.0
Overall token-F1	73.9	89.4	+15.6
Answerable Exact Match	15.0	55.0	+40.0
Answerable token-F1	50.2	81.4	+31.1
Not-found abstention acc	97.5	97.5	+0.0

The base model already abstains well on absent clauses; the fine-tune's main gain is returning exact clause spans instead of verbose paraphrases, with no regression on the not-found class.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "unsloth/gemma-4-E4B-it"
ADAPTER = "soybelli/gemma-4-E4B-it-cuad-lora"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

SYSTEM = (
    "You are a legal contract review assistant. Read the contract excerpt and answer the question. "
    "If the relevant clause is present, quote the exact text from the excerpt. "
    "If it is not present, reply exactly: Not found."
)
excerpt = "...THIS AGREEMENT shall be governed by and construed in accordance with the laws of the State of New York..."
question = (
    'Highlight the parts (if any) of this contract related to "Governing Law" that should be reviewed by a '
    "lawyer. Details: Which state/country's law governs the interpretation of the contract?"
)
prompt = f"{SYSTEM}\n\n### Contract excerpt:\n{excerpt}\n\n### Question:\n{question}\n\n### Answer:"

messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Training

• Data: CUAD SQuAD-2.0 JSON (CUAD_v1/CUAD_v1.json), flattened to (excerpt, question, answer) pairs. Excerpts are ~2,400-char windows centered on the answer span; the natural ~32% answerable / 68% not-found distribution is kept. 18,759 training examples (10% of contracts held out for eval).
• Config: 1 epoch, max_seq_length=1024, effective batch 16, lr 2e-4, adamw_8bit, bf16, trained only on the assistant response. Final train loss ≈ 0.14.
• Hardware: single NVIDIA RTX 5090 (~95 min).

Limitations

• Trained on ~2,400-char (≈600-token) excerpts; for full contracts, chunk the document and query each chunk.
• English commercial contracts only; not legal advice.
• Inherits the base model's and CUAD's biases. CUAD is licensed CC-BY-4.0.