qwen2.5-3b-gec-v2

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for English grammatical error correction (GEC). v2 improvements over qwen2.5-3b-gec-bea2019:

• +0.038 F0.5 (0.495 → 0.533) on held-out BEA-dev.
• Mixed training corpus: BEA-2019 W&I+LOCNESS (22,668 pairs) + Grammarly Coedit GEC subset (19,823 pairs) = 42,491 minimal-edit pairs.
• All 36 transformer layers LoRA-adapted (vs 16 in v1) → 14.97 M trainable params (vs 6.65 M).
• 5,000 iters (vs 3,000) on the larger corpus.
• LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

Results

Setting	Precision	Recall	F0.5
v2 (post-processed, single-pass)	0.589	0.386	0.533
v2 + iterative (max-iter 2)	0.575	0.407	0.531
v1 (BEA only)	0.543	0.365	0.495

Scored with ERRANT on a 100-sample held-out BEA-dev split (ABCN.dev.gold.bea19.m2, identity rows skipped). Post-processing strips a small set of BEA-2019 tokenization artifacts (trailing " ." → ".", collapsed double spaces, source-aligned leading-cap).

Validation loss trajectory (held-out 500-pair mix of BEA + Coedit):

Iter	Val
1 (baseline)	1.981
500	0.367
1000	0.297
1500	0.319
2000	0.251
2500	0.260
3000	0.231
3500	0.160 (best, promoted)
4000	0.248
4500	0.248
5000	0.167

Trained 5,000 iters on an Apple M2 Max in ~36 minutes (peak memory 10.5 GB).

Use

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-gec-v2")
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "Correct the grammar of the user text. Preserve meaning."},
        {"role": "user", "content": "He go to school"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=80))
# -> "He goes to school"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-gec-v2") works the same.

System prompt

Correct the grammar of the user text. Preserve meaning.

Use that exact prompt for in-distribution behavior. Chat template is Qwen's ChatML.

Recommended decoding

• Greedy (temperature 0).
• max_new_tokens ~ 1.5× the prompt length.
• Single-pass: ~80 tokens / sentence on M2 Max greedy.
• Iterative re-feed: same F0.5 on this dataset (trades precision for recall); add it if you need higher recall.
• Post-processor (~20 LOC, in the source repo) trims " .", collapses runs of whitespace, and aligns leading capitalisation to the input. Worth +0.01–0.02 F0.5.

Training

• Base: Qwen/Qwen2.5-3B-Instruct (3.09 B params)
• LoRA: rank 16, alpha 32, all 36 layers (vs 16 in v1), 14.97 M trainable (0.485 %)
• Optimizer: AdamW, lr 5e-5
• Batch size: 2, max sequence 256
• Iters: 5,000 (best snapshot at iter 3,500 was promoted)
• Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
• Data:
  • BEA-2019 W&I+LOCNESS ABC.train.gold.bea19.m2 → 22,668 unique src→tgt pairs after dropping identity rows.
  • Grammarly Coedit train.jsonl, filtered to task == "gec" → 19,823 pairs after stripping instruction prefixes.
  • 500 held-out for val.

Limitations

• ERRANT F0.5 = 0.533 is below the GECToR paper's 0.65+ benchmark. Closing the gap further requires NUCLE + FCE + Lang-8 + the multi-stage curriculum (synthetic pretrain → BEA mid-train → W&I+L finetune).
• Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful corrections.
• English only.
• Trained on ≤ 256-token sequences; long inputs may degrade.

Comparison to v1

The v1 model is BEA-only and uses a smaller LoRA (16 layers). v2 doubles LoRA capacity and adds Coedit data; the gains come about evenly from both. Use v2 unless you specifically want a smaller-footprint adapter.

Citation

@misc{qwen25_3b_gec_v2,
  title  = {qwen2.5-3b-gec-v2: LoRA-tuned Qwen2.5-3B for English GEC},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-gec-v2}
}

Datasets: Bryant et al. The BEA-2019 Shared Task on GEC. Raheja et al. CoEdIT: Text Editing by Task-Specific Instruction Tuning.