--- license: apache-2.0 language: - en - ro - pl - hu base_model: Qwen/Qwen3-4B library_name: transformers pipeline_tag: text-generation tags: - lora - on-device - mlx - legal - consumer-contracts - red-flag-detection - flowx --- # caveat **caveat** is a small, on-device model that reads a consumer contract and flags the clauses worth reading twice: auto-renewal traps, punitive early-termination, liability exclusions, data-selling, arbitration / class-action waivers, and the rest of a 34-type taxonomy. It runs **privately on-device** (no API, no data leaves the machine) in **English, Romanian, Polish and Hungarian**, and returns structured JSON, not prose. > **caveat explains consequences. It is NOT legal advice and does not tell you whether to > sign.** A no-advice contract is enforced at generation time (the model is scanned for > advice language; violations are rejected). - **Base model:** [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) (Apache-2.0), LoRA fine-tuned. - **Deployment:** on-device. Ship artifact is **MLX int4** (`mlx-int4/`, ~2 GB, Apple Silicon); MLX int8 (`mlx-int8/`) and the standard fp16 merged model (repo root) are also provided. - **Full model card:** see [`caveat_model_card.pdf`](./caveat_model_card.pdf). ## What it does Input: one contract chunk + a document-type hint. Output: a JSON `ChunkResult`: ```json { "summary": "", "red_flags": [ {"clause_type": "auto_renewal", "severity": "warning", "span_text": "", "explanation": "", "truncated": false} ] } ``` `clause_type` is one of 34 taxonomy ids across 6 categories (money, rights-waived, time/renewal, lease, insurance, data); `severity` is `info | caution | warning`. ### Recommended inference: two passes, unioned Real contracts pack many red flags into few paragraphs, and a whole-document pass and a per-paragraph pass surface **different** clauses (insurance detects better with full-document context; lease/loan clauses detect better isolated). Run both and union the results (de-duplicate by clause type + span). This roughly doubles real-world recall versus a single whole-document pass. See `demo/explain.py` in the project repo for a reference implementation. ## Intended use **In scope:** consumer-facing contracts (subscriptions / terms of service, insurance policies, loan / credit agreements, residential leases, utility / telecom) in EN/RO/PL/HU, as a **triage aid** that points a human to clauses worth a closer look. **Out of scope:** legal advice or a decision on whether to sign; B2B / M&A / commercial contracts; jurisdictions or languages outside those trained; any use where a missed clause carries safety or legal risk without human review. Recall is partial (see Limitations). ## Evaluation **Real-world recall (the headline).** On five held-out real consumer contracts (gym EN, home-insurance EN, personal-loan RO, apartment-lease PL, mobile HU), scored with the two-pass inference, caveat detects **13 distinct red flags**, up from 9 for a prior 4B and 7 for a 1.7B baseline, and it is the first version to catch flags on **all five** documents (the RO loan and PL lease were previously undetected). **Synthetic benchmark (RedFlag-Bench v0.1, 597 frozen held-out chunks).** Overall clause-type F1 **0.275** (+54% over the pre-coverage-fix 4B); by category: lease 0.33, money 0.32, data 0.32, rights-waived 0.23, time/renewal 0.20, insurance 0.00\*; by language: EN 0.35, HU 0.29, PL 0.24, RO 0.14. Faithfulness: 2.6 span-verification failures per 100 flags; JSON validity 1.00; advice hits 0. *(\*The insurance bench F1 is a small-sample artifact of the frozen set; real-document insurance detection works, e.g. waiting-periods and buried-exclusions on the home-insurance doc.)* The synthetic bench shares a generator family with the training data, so read it as a format-adherence / faithfulness ceiling, not a real-world recall measure. **Versus a frontier model (like-for-like slice).** On a seeded 100-chunk slice, a frontier model (Claude Opus) leads the overall synthetic F1 (0.385 vs caveat 0.233); caveat wins specific cells (lease 0.80 vs 0.30, English 0.63 vs 0.27, severity agreement 0.43 vs 0.33). caveat's value is real-document recall and running **privately, on-device, at no inference cost**, not beating a frontier model on the synthetic benchmark. ## Training LoRA (rank 16 / alpha 32 / dropout 0.05, all-linear target modules), 2 epochs, cosine LR 2e-5 -> 2e-6, effective batch 4, max sequence length 2048, bf16, on an NVIDIA L4. Training data (5,263 train / 1,367 validation chunks, split by document): synthetic consumer contracts generated across five drafting registers with flag-dense examples, targeted insurance- and lease-coverage batches, and **450 real English clause spans from CUAD** (train-only). ### Attribution & licenses - Model weights released under **Apache-2.0** (inherited from the Qwen3-4B base). - Training used the **Contract Understanding Atticus Dataset (CUAD) v1**, The Atticus Project, Inc. (CC-BY-4.0) — Hendrycks et al., *CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review*, arXiv:[2103.06268](https://arxiv.org/abs/2103.06268). CUAD clause spans were used train-only for clause shapes. ## Limitations & responsible use - **Not legal advice.** A triage aid only; always have important contracts reviewed by a qualified professional. - **Partial recall.** caveat surfaces many but not all red flags (~13 of 20+ on the real set); a clean result is not a guarantee the contract is safe. - **Uneven coverage.** English is strongest; Romanian and Polish are weaker; insurance and lease phrasing vary in reliability. Use the two-pass inference for best recall. - **Small real-world eval.** The real-document evidence is five contracts; treat the numbers as directional. ## Citation ```bibtex @misc{flowx_caveat_2026, title = {caveat: on-device consumer-contract red-flag detection}, author = {R\u{a}du\c{t}\u{a}, Bogdan}, year = {2026}, note = {FlowX.AI. LoRA fine-tune of Qwen3-4B.} } ``` _Author: Bogdan Răduță, Head of Research, FlowX.AI._