---
language:
- or
license: apache-2.0
pipeline_tag: image-to-text
tags:
- odia
- ocr
- vision-language
- qwen2-vl
- peft
- lora
- image-to-text
- optical-character-recognition
datasets:
- shantipriya/odia-ocr-merged
base_model: Qwen/Qwen2.5-VL-3B-Instruct
model-index:
- name: odia-ocr-qwen-finetuned_v2
  results:
  - task:
      type: image-to-text
      name: Odia OCR
    dataset:
      name: shantipriya/odia-ocr-merged
      type: shantipriya/odia-ocr-merged
    metrics:
    - type: word_accuracy
      name: Word Accuracy (printed)
      value: 65-70
    - type: cer
      name: CER estimate
      value: "<0.15"
---

# Odia OCR — Qwen2.5-VL-3B Fine-tuned (v2)

Fine-tuned version of [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) for **Optical Character Recognition (OCR) of Odia script** using LoRA adapters trained on 145,000 word-level Odia image crops.

🔗 **Mirror (OdiaGenAI org):** [OdiaGenAIOCR/odia-ocr-qwen-finetuned_v2](https://huggingface.co/OdiaGenAIOCR/odia-ocr-qwen-finetuned_v2)  
🔗 **Merged model (no PEFT):** See [`merged/`](#merged-full-model-no-peft-required) subfolder or [OdiaGenAIOCR/odia-ocr-qwen-finetuned-merged](https://huggingface.co/OdiaGenAIOCR/odia-ocr-qwen-finetuned-merged)

---

## Quick Start

### Option 1 — LoRA Adapter (with PEFT)

```python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
import torch
from PIL import Image

base_model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_id    = "shantipriya/odia-ocr-qwen-finetuned_v2"

processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model_id, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

def ocr_odia(image_path: str) -> str:
    image = Image.open(image_path).convert("RGB")
    messages = [{
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text",  "text": "Extract the Odia text from this image. Return only the text."},
        ],
    }]
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
    return processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0].strip()

print(ocr_odia("odia_word.png"))
```

### Option 2 — Merged Model (no PEFT required)

```python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
import torch
from PIL import Image

model_id  = "shantipriya/odia-ocr-qwen-finetuned_v2"
subfolder = "merged"

processor = AutoProcessor.from_pretrained(model_id, subfolder=subfolder)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id, subfolder=subfolder, torch_dtype=torch.float16, device_map="auto"
)
model.eval()

image = Image.open("odia_text.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text",  "text": "Extract all Odia text from this image. Return only the text."},
    ],
}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
result = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(result)
```

### Option 3 — HF Serverless Inference API

```python
import base64, io, requests
from PIL import Image

token    = "hf_..."   # your HuggingFace token
API_URL  = "https://router.huggingface.co/v1/chat/completions"
MODEL_ID = "OdiaGenAIOCR/odia-ocr-qwen-finetuned"   # hosted serverless copy

def img_to_b64(image: Image.Image) -> str:
    buf = io.BytesIO()
    image.save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode()

image = Image.open("odia_word.png").convert("RGB")

payload = {
    "model": MODEL_ID,
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the Odia text from this image. Return only the text."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_to_b64(image)}"}},
        ],
    }],
    "max_tokens": 128,
    "temperature": 0,
}

resp = requests.post(API_URL, headers={"Authorization": f"Bearer {token}"}, json=payload)
print(resp.json()["choices"][0]["message"]["content"])
```

### Option 4 — 4-bit Quantized (low VRAM, requires bitsandbytes)

```python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, "shantipriya/odia-ocr-qwen-finetuned_v2")
model.eval()
```

---

## Model Details

| Property | Value |
|----------|-------|
| **Base Model** | Qwen/Qwen2.5-VL-3B-Instruct |
| **Fine-tuning Method** | LoRA (PEFT) |
| **LoRA Rank** | 64 |
| **LoRA Alpha** | 128 |
| **LoRA Target Modules** | q\_proj, v\_proj |
| **Training Dataset** | shantipriya/odia-ocr-merged |
| **Training Samples** | 145,000 word-level Odia OCR crops |
| **Final Checkpoint** | checkpoint-6400 |
| **Final Epoch** | 1.50 |
| **Final Train Loss** | ~4.83 |
| **Best Eval Loss** | 5.454 |
| **Training Hardware** | NVIDIA H100 80GB |
| **Training Duration** | ~12.7 hours |
| **Learning Rate** | 3e-4 (cosine decay to 2.7e-5) |
| **Batch Size** | 8 (per device 2 × grad accum 4) |

---

## Evaluation

Qualitative evaluation on ~200 held-out Odia word-level crops:

| Category | Share | Description |
|----------|:-----:|---|
| ✅ Exact match | ~65–70% | Clean, well-segmented printed crops |
| ⚠️ 1–2 char error | ~20–25% | Complex conjuncts, long-vowel matras |
| ❌ Mostly incorrect | ~10–15% | Degraded scans, long compound words, low-res |

> Full CER/WER metrics on a curated test split are pending. See [BENCHMARK_REPORT](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2) for latest results.

---

## Sample Predictions

### ✅ Good Predictions — clean, high-contrast printed crops

| Image | Ground Truth | Extracted | Remark |
|:---:|:---:|:---:|---|
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/good_01.png) | ଓଡ଼ିଆ | ଓଡ଼ିଆ | ✅ Exact match |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/good_02.png) | ସରକାର | ସରକାର | ✅ Exact match |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/good_03.png) | ଭାରତ | ଭାରତ | ✅ Exact match |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/good_04.png) | ବିଦ୍ୟାଳୟ | ବିଦ୍ୟାଳୟ | ✅ Exact match |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/good_05.png) | ନମସ୍କାର | ନମସ୍କାର | ✅ Exact match |

### ⚠️ Mixed Predictions — partial errors

| Image | Ground Truth | Extracted | Remark |
|:---:|:---:|:---:|---|
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/mixed_01.png) | ସ୍ୱାଧୀନତା | ସ୍ୱାଦୀନତା | ⚠️ Diacritic substitution (ଧ→ଦ) |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/mixed_02.png) | ପ୍ରତିଷ୍ଠା | ପ୍ରତିଷ୍ଟା | ⚠️ Conjunct error (ଷ୍ଠ→ଷ୍ଟ) |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/mixed_03.png) | ଅନୁଷ୍ଠାନ | ଅନୁଷ୍ଟାନ | ⚠️ Halant confusion in conjunct |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/mixed_04.png) | ଦୃଷ୍ଟିଭଙ୍ଗୀ | ଦୃଷ୍ଟିଭଙ୍ଗି | ⚠️ Final vowel matra dropped |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/mixed_05.png) | ଶ୍ରେଣୀ | ଶ୍ରେଣି | ⚠️ Vowel length confusion (ୀ→ି) |

### ❌ Bad Predictions — degraded or complex crops

| Image | Ground Truth | Extracted | Remark |
|:---:|:---:|:---:|---|
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/bad_01.png) | ଉତ୍ପାଦନ | ଉପ୍ରାଦ | ❌ Multiple substitutions |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/bad_02.png) | ବ୍ୟବସ୍ଥାପନା | ବ୍ୟବ | ❌ Truncated — sequence too long |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/bad_03.png) | ମହାବିଦ୍ୟାଳୟ | ମହାବିଦ | ❌ Long compound word truncation |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/bad_04.png) | ସ୍ୱାଭାବିକ | ସ୍ୱଭ | ❌ Heavy degradation, partial output |
| ![OCR sample](https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2/resolve/main/assets/samples/bad_05.png) | ଅଧ୍ୟାପକ | ଆଦ୍ୟାପ | ❌ Vowel-initial confusion |

---

## Training Notes

Training was **early stopped at step 6,400** (of 12,387 planned) due to confirmed loss plateau:
- Train loss converged to ~4.83–5.0 by step ~800 with no further improvement
- Gradient norms remained tiny (~0.014–0.024) indicating saturated word-level learning
- Eval loss plateau: 5.512 → 5.454 (only 1% delta across 6,000 steps)

For further gains, Phase 3 with mixed paragraph + word samples is recommended.

---

## Available Checkpoints

| Checkpoint | Step | Epoch | Train Loss |
|-----------|------|-------|-----------|
| [checkpoint-3200](checkpoint-3200/) | 3,200 | 0.77 | ~5.2 |
| [checkpoint-6000](checkpoint-6000/) | 6,000 | 1.45 | ~4.85 |
| [checkpoint-6200](checkpoint-6200/) | 6,200 | 1.50 | ~4.92 |
| **[checkpoint-6400](checkpoint-6400/)** ← **Final** | 6,400 | 1.51 | ~4.83 |

---

## Merged Full Model (No PEFT Required)

A fully merged model (`base + LoRA → single safetensors`) is in the `merged/` subfolder — **no PEFT library** needed:

| File | Size |
|------|------|
| `merged/model.safetensors` | 7.0 GB |
| `merged/config.json` | — |
| `merged/tokenizer.json` | — |
| `merged/tokenizer_config.json` | — |
| `merged/vocab.json` | — |
| `merged/merges.txt` | — |
| `merged/preprocessor_config.json` | — |
| `merged/processor_config.json` | — |
| `merged/generation_config.json` | — |
| `merged/chat_template.jinja` | — |

See **Option 2** in the Quick Start section above.  The merged model is also separately mirrored at [OdiaGenAIOCR/odia-ocr-qwen-finetuned-merged](https://huggingface.co/OdiaGenAIOCR/odia-ocr-qwen-finetuned-merged).

---

## Training Data

Trained on [shantipriya/odia-ocr-merged](https://huggingface.co/datasets/shantipriya/odia-ocr-merged):
- **145,000** word-level Odia script image crops
- Diverse fonts, sizes, and print quality
- Sourced from multiple Odia OCR corpora and merged/deduplicated

---

## Limitations

- Optimised for **printed Odia word-level crops**; handwritten or heavily degraded text needs additional fine-tuning
- Complex conjunct characters and long compound words are the main error sources
- Not tested on mixed-language (Odia + English) documents
- For full-page / paragraph OCR, tile the image into ~300 px strips and concatenate outputs

---

## Citation

```bibtex
@misc{parida2026odiaocr,
  author       = {Shantipriya Parida},
  title        = {Odia OCR: Fine-tuned Qwen2.5-VL for Odia Script Recognition},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v2}},
  note         = {LoRA fine-tune of Qwen2.5-VL-3B-Instruct on 145K Odia OCR word crops}
}
```

If using the training dataset, also cite:

```bibtex
@misc{parida2026odiadataset,
  author       = {Shantipriya Parida},
  title        = {Odia OCR Merged Dataset},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/datasets/shantipriya/odia-ocr-merged}}
}
```

---

## License

Apache 2.0

## Contact

- **Author**: Shantipriya Parida    
- **Mirror**: [OdiaGenAIOCR/odia-ocr-qwen-finetuned_v2](https://huggingface.co/OdiaGenAIOCR/odia-ocr-qwen-finetuned_v2)
<!-- updated: 1772029878 -->