---
language:
- uk
library_name: transformers
pipeline_tag: image-to-text
tags:
- trocr
- vision-encoder-decoder
- handwritten-text-recognition
- ocr
- ukrainian
- rukopys
base_model: kazars24/trocr-base-handwritten-ru
datasets:
- UkrainianCatholicUniversity/rukopys
metrics:
- cer
- wer
model-index:
- name: trocr-rukopys
  results:
  - task:
      type: image-to-text
      name: Handwritten Text Recognition
    dataset:
      type: UkrainianCatholicUniversity/rukopys
      name: Rukopys, gold val
      split: validation
    metrics:
    - type: cer
      value: 0.1146
      name: CER
    - type: wer
      value: 0.3089
      name: WER
---

# trocr-rukopys

A **TrOCR-base** (`VisionEncoderDecoderModel`, ~334M params) line recognizer for
handwritten Ukrainian document text, fine-tuned on the
[Rukopys](https://huggingface.co/datasets/UkrainianCatholicUniversity/rukopys) dataset.

The training data included `handwritten`, `printed`, `annotation` and `table` region crops from Rukopys.

## TL;DR

| | value |
|---|---|
| Architecture | TrOCR-base (ViT-B/16 encoder + TrOCR decoder), `VisionEncoderDecoderModel` |
| Parameters | ~334M |
| Init from | [`kazars24/trocr-base-handwritten-ru`](https://huggingface.co/kazars24/trocr-base-handwritten-ru) |
| Handles | `handwritten`, `printed`, `table`, `annotation` |
| Gold-val CER / WER | **0.1146 / 0.3089** |
| Input | a single cropped text region/line (RGB) |
| Output | the transcribed string (metric-normalized character set) |

## Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream
of a layout detector.

For a stronger recognizer from a later, larger training generation, see
[`Hukyl/trocr-large-rukopys`](https://huggingface.co/Hukyl/trocr-large-rukopys).

## How to use

```python
from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-rukopys"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")          # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)
```

## Lineage & training curriculum

Two-stage silver→gold fine-tuning from a Russian handwriting base. Optimizer: AdamW via
the 🤗 `Seq2SeqTrainer`, linear LR decay, `max_target_length` 256, online augmentation,
seed 42.

1. **Base** — [`kazars24/trocr-base-handwritten-ru`](https://huggingface.co/kazars24/trocr-base-handwritten-ru):
   `microsoft/trocr-base-handwritten` fine-tuned for Russian handwriting.
2. **Silver pretrain** (5 ep): partial-freeze pretrain on the Rukopys *silver* split
   (~119.8k auto-labeled crops).
3. **Gold fine-tune** (up to 15 ep, early stopping with patience 3): full
   encoder + decoder unfreeze on the human-labeled *gold* train split,
   19,069 train / 3,366 val crops, stratified at crop level.

### Hyperparameters (as launched)

| hyperparameter | value |
|---|---|
| epochs (max) | 15 (early stopping, patience 3) |
| batch size | 32 |
| learning rate | 1e-4 |
| warmup ratio | 0.1 |
| max target length | 256 tokens |
| optimizer / schedule | AdamW, linear decay |
| online augmentation | on (`default` profile) |
| eval | greedy (`num_beams=1`), eval batch 64 |
| precision / device | bf16, CUDA |
| seed | 42 |

The full recorded recipe ships in this repo's `training_meta.json`.

### Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by _one geometric_ + _one or two photometric_ operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

| pool | handwritten / annotation | printed |
|---|---|---|
| geometric (pick 1) | margin pad/trim, rotation ±1–5°, elastic distortion, baseline warp | margin pad/trim, rotation ±1–3° |
| photometric (pick 1–2) | paper colour shift, Gaussian noise, JPEG recompression (q30–65), contrast/gamma, morphological op | paper colour shift, Gaussian noise, JPEG recompression (q40–70), contrast/gamma, morphological op |

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting
deformation, widening the range of appearances beyond the raw training crops.

## Results

Protocol: greedy decode, `max_new_tokens=512`, 1000-sample draw from the gold
validation split, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV
canonicalized) so CER/WER reflect the scored character set.

### Overall

| metric | value |
|---|---|
| CER | 0.1146 |
| WER | 0.3089 |
| exact-match accuracy | 0.3830 |
| n_samples | 1000 |

### Per class

| class | CER | WER | accuracy | n |
|---|---|---|---|---|
| handwritten | 0.1126 | 0.3036 | 0.3841 | 958 |
| printed | 0.0987 | 0.2826 | 0.4167 | 12 |
| table | 0.1915 | 0.5000 | 0.5000 | 2 |
| annotation | 0.4278 | 0.9268 | 0.3214 | 28 |

We also acknowledge that `printed`/`table`/`annotation` n is quite small, so measuring CER against them is quite noisy.

## Limitations & biases

- Handwritten Ukrainian archival document material only; expect degradation on other
  scripts, languages, or modern born-digital text.
- The character set and normalization are tuned to the document-OCR metric — outputs are
  normalized text, not faithful transcription.
- The train/val split is stratified at crop level, not page level — crops from the same
  page (and writer) appear on both sides, so the reported val CER is optimistic compared
  to a page-level protocol.
- `annotation` and `table` classes are weak.
- Single seed and validation split — no across-run variance estimate.

## Training data & attribution

| dataset | source | license | role |
|---|---|---|---|
| Rukopys | [`UkrainianCatholicUniversity/rukopys`](https://huggingface.co/datasets/UkrainianCatholicUniversity/rukopys) | CC BY 4.0 | silver pretrain (auto-labeled) + gold fine-tune |

The base model `kazars24/trocr-base-handwritten-ru` declares no explicit license — verify
upstream terms before redistribution.