---
license: apache-2.0
base_model: Qwen/Qwen2.5-3B-Instruct
tags:
- grammar-correction
- gec
- bea-2019
- lora
- mlx
- english
language:
- en
pipeline_tag: text-generation
---

# qwen2.5-3b-gec-bea2019

LoRA fine-tune of `Qwen/Qwen2.5-3B-Instruct` for English grammatical error correction (GEC). Trained on the **W&I+LOCNESS** subset of BEA-2019 (33,432 minimal-edit pairs). LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

## Results

ERRANT scores on a 100-sample held-out BEA-dev split (sampled from `ABCN.dev.gold.bea19.m2`):

| Metric | Value |
|---|---|
| **Precision** | 0.543 |
| **Recall** | 0.365 |
| **F0.5** | **0.4950** |

Validation loss converged from 2.875 (baseline) to 0.281 over 3,000 LoRA iters on an Apple M2 Max in ~17 minutes.

## Use

```python
from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-gec-bea2019")
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "Correct the grammar of the user text. Preserve meaning."},
        {"role": "user", "content": "He go to school"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=80))
# -> "He goes to school"
```

PyTorch users: `transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-gec-bea2019")` works the same way.

## System prompt

The model was trained with the system prompt:

```
Correct the grammar of the user text. Preserve meaning.
```

Use that exact prompt for in-distribution behavior. The chat template is Qwen's ChatML.

## Smoke samples

| Input | Output |
|---|---|
| `He go to school` | `He goes to school` |
| `I are happy` | `I am happy` |
| `she walk fast` | `She walks fast .` |
| `they was tired yesterday` | `They were tired yesterday .` |
| `she have a cat` | `She has a cat .` |

Trailing-space + period is a BEA-2019 tokenization artifact; a post-processor can strip it.

## Training

- **Base**: `Qwen/Qwen2.5-3B-Instruct` (3.09 B params)
- **LoRA**: rank 16, alpha 32, 16 layers, 6.65 M trainable params (0.216 %)
- **Optimizer**: AdamW, lr 5e-5
- **Batch size**: 2, max sequence 256
- **Iters**: 3,000
- **Hardware**: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
- **Data**: BEA-2019 W&I+LOCNESS `ABC.train.gold.bea19.m2`, converted to ChatML SFT by the [data-pipeline](https://github.com/plutonium-guy/grammer_correction_model) repo. 33,432 minimal-edit (src, tgt) pairs after dropping identity rows.
- **Wall-clock**: 17 minutes solo (3.2 it/s).

## Limitations

- F0.5 = 0.4950 is below the GECToR paper's 0.65+ benchmark. That gap is expected: the published number comes from a multi-stage curriculum (C4_200M pretrain → BEA-2019 → W&I+L fine-tune) on a token-classification head. This run is a single LoRA pass with seq2seq generation. Closing the gap needs more data (NUCLE, FCE, Lang-8) and the multi-stage recipe.
- Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful corrections.
- English only.
- Trained on ≤ 256-token sequences; long inputs may degrade.

## Citation

```bibtex
@misc{qwen25_3b_gec_bea2019,
  title  = {qwen2.5-3b-gec-bea2019: LoRA-tuned Qwen2.5-3B for English GEC},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-gec-bea2019}
}
```

Underlying corpus: Bryant, Felice, Andersen, Briscoe. *The BEA-2019 Shared Task on Grammatical Error Correction*, BEA 2019.