README.md

---
license: apache-2.0
language:
  - en
  - ro
  - pl
  - hu
base_model: Qwen/Qwen3-4B
library_name: transformers
pipeline_tag: text-generation
tags:
  - lora
  - on-device
  - mlx
  - legal
  - consumer-contracts
  - red-flag-detection
  - flowx
---

# caveat

**caveat** is a small, on-device model that reads a consumer contract and flags the
clauses worth reading twice: auto-renewal traps, punitive early-termination, liability
exclusions, data-selling, arbitration / class-action waivers, and the rest of a 34-type
taxonomy. It runs **privately on-device** (no API, no data leaves the machine) in
**English, Romanian, Polish and Hungarian**, and returns structured JSON, not prose.

> **caveat explains consequences. It is NOT legal advice and does not tell you whether to
> sign.** A no-advice contract is enforced at generation time (the model is scanned for
> advice language; violations are rejected).

- **Base model:** [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) (Apache-2.0), LoRA fine-tuned.
- **Deployment:** on-device. Ship artifact is **MLX int4** (`mlx-int4/`, ~2 GB, Apple Silicon);
  MLX int8 (`mlx-int8/`) and the standard fp16 merged model (repo root) are also provided.
- **Full model card:** see [`caveat_model_card.pdf`](./caveat_model_card.pdf).

## What it does

Input: one contract chunk + a document-type hint. Output: a JSON `ChunkResult`:

```json
{
  "summary": "<one plain-language sentence>",
  "red_flags": [
    {"clause_type": "auto_renewal", "severity": "warning",
     "span_text": "<verbatim clause text>", "explanation": "<consequence, no advice>",
     "truncated": false}
  ]
}
```

`clause_type` is one of 34 taxonomy ids across 6 categories (money, rights-waived,
time/renewal, lease, insurance, data); `severity` is `info | caution | warning`.

### Recommended inference: two passes, unioned

Real contracts pack many red flags into few paragraphs, and a whole-document pass and a
per-paragraph pass surface **different** clauses (insurance detects better with full-document
context; lease/loan clauses detect better isolated). Run both and union the results
(de-duplicate by clause type + span). This roughly doubles real-world recall versus a single
whole-document pass. See `demo/explain.py` in the project repo for a reference implementation.

## Intended use

**In scope:** consumer-facing contracts (subscriptions / terms of service, insurance
policies, loan / credit agreements, residential leases, utility / telecom) in EN/RO/PL/HU,
as a **triage aid** that points a human to clauses worth a closer look.

**Out of scope:** legal advice or a decision on whether to sign; B2B / M&A / commercial
contracts; jurisdictions or languages outside those trained; any use where a missed clause
carries safety or legal risk without human review. Recall is partial (see Limitations).

## Evaluation

**Real-world recall (the headline).** On five held-out real consumer contracts (gym EN,
home-insurance EN, personal-loan RO, apartment-lease PL, mobile HU), scored with the
two-pass inference, caveat detects **13 distinct red flags**, up from 9 for a prior 4B and 7
for a 1.7B baseline, and it is the first version to catch flags on **all five** documents
(the RO loan and PL lease were previously undetected).

**Synthetic benchmark (RedFlag-Bench v0.1, 597 frozen held-out chunks).** Overall
clause-type F1 **0.275** (+54% over the pre-coverage-fix 4B); by category: lease 0.33,
money 0.32, data 0.32, rights-waived 0.23, time/renewal 0.20, insurance 0.00\*; by language:
EN 0.35, HU 0.29, PL 0.24, RO 0.14. Faithfulness: 2.6 span-verification failures per 100
flags; JSON validity 1.00; advice hits 0. *(\*The insurance bench F1 is a small-sample
artifact of the frozen set; real-document insurance detection works, e.g. waiting-periods and
buried-exclusions on the home-insurance doc.)* The synthetic bench shares a generator family
with the training data, so read it as a format-adherence / faithfulness ceiling, not a
real-world recall measure.

**Versus a frontier model (like-for-like slice).** On a seeded 100-chunk slice, a frontier
model (Claude Opus) leads the overall synthetic F1 (0.385 vs caveat 0.233); caveat wins
specific cells (lease 0.80 vs 0.30, English 0.63 vs 0.27, severity agreement 0.43 vs 0.33).
caveat's value is real-document recall and running **privately, on-device, at no inference
cost**, not beating a frontier model on the synthetic benchmark.

## Training

LoRA (rank 16 / alpha 32 / dropout 0.05, all-linear target modules), 2 epochs, cosine LR
2e-5 -> 2e-6, effective batch 4, max sequence length 2048, bf16, on an NVIDIA L4. Training
data (5,263 train / 1,367 validation chunks, split by document): synthetic consumer contracts
generated across five drafting registers with flag-dense examples, targeted insurance- and
lease-coverage batches, and **450 real English clause spans from CUAD** (train-only).

### Attribution & licenses

- Model weights released under **Apache-2.0** (inherited from the Qwen3-4B base).
- Training used the **Contract Understanding Atticus Dataset (CUAD) v1**, The Atticus
  Project, Inc. (CC-BY-4.0) — Hendrycks et al., *CUAD: An Expert-Annotated NLP Dataset for
  Legal Contract Review*, arXiv:[2103.06268](https://arxiv.org/abs/2103.06268). CUAD clause
  spans were used train-only for clause shapes.

## Limitations & responsible use

- **Not legal advice.** A triage aid only; always have important contracts reviewed by a
  qualified professional.
- **Partial recall.** caveat surfaces many but not all red flags (~13 of 20+ on the real set);
  a clean result is not a guarantee the contract is safe.
- **Uneven coverage.** English is strongest; Romanian and Polish are weaker; insurance and
  lease phrasing vary in reliability. Use the two-pass inference for best recall.
- **Small real-world eval.** The real-document evidence is five contracts; treat the numbers
  as directional.

## Citation

```bibtex
@misc{flowx_caveat_2026,
  title  = {caveat: on-device consumer-contract red-flag detection},
  author = {R\u{a}du\c{t}\u{a}, Bogdan},
  year   = {2026},
  note   = {FlowX.AI. LoRA fine-tune of Qwen3-4B.}
}
```

_Author: Bogdan Răduță, Head of Research, FlowX.AI._