---
language:
- uk
library_name: transformers
pipeline_tag: object-detection
license: apache-2.0
base_model: docling-project/docling-layout-heron-101
datasets:
- UkrainianCatholicUniversity/rukopys
metrics:
- precision
- recall
tags:
- rt-detr
- rt-detrv2
- object-detection
- document-layout-analysis
- handwritten
- ukrainian
- rukopys
- docling
model-index:
- name: rt-detrv2-heron-101-rukopys
  results:
  - task:
      type: object-detection
      name: Document Layout Detection
    dataset:
      type: UkrainianCatholicUniversity/rukopys
      name: Rukopys, gold val
      split: validation
    metrics:
    - type: mAP@50
      value: 0.7685
      name: mAP@50
    - type: mAP@50-95
      value: 0.4819
      name: mAP@50-95
---

# rt-detrv2-heron-101-rukopys

An **RT-DETRv2-R101** (`RTDetrV2ForObjectDetection`, ~76.7M params)
document-layout detector for handwritten Ukrainian pages, fine-tuned on the
[Rukopys](https://huggingface.co/datasets/UkrainianCatholicUniversity/rukopys)
dataset with a 7-class head:
`0 handwritten · 1 printed · 2 formula · 3 table · 4 annotation · 5 image · 6 graph`.

## TL;DR

| | value |
|---|---|
| Architecture | RT-DETRv2, ResNet-101 backbone, `RTDetrV2ForObjectDetection` |
| Parameters | ~76.7M |
| Init from | [`docling-project/docling-layout-heron-101`](https://huggingface.co/docling-project/docling-layout-heron-101) (17-class document-layout checkpoint) |
| Classes | `handwritten`, `printed`, `formula`, `table`, `annotation`, `image`, `graph` |
| Decoding | set prediction, 300 object queries, no NMS |
| Gold-val mAP@50 / mAP@50-95 | **0.7685 / 0.4819** (macro over 7 classes) |
| Input | a document page image, 640×640 |
| Output | class-labeled region bounding boxes |

## Intended use

Detecting and classifying page regions on handwritten Ukrainian documents, upstream
of region recognizers (e.g.
[`Hukyl/trocr-large-rukopys`](https://huggingface.co/Hukyl/trocr-large-rukopys) for
text, [`Hukyl/trocr-base-rukopys-formula`](https://huggingface.co/Hukyl/trocr-base-rukopys-formula)
for formulas). For a stronger single detector on the same task, see
[`Hukyl/doclayout-yolov10-rukopys`](https://huggingface.co/Hukyl/doclayout-yolov10-rukopys).

## How to use

```python
import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

repo = "Hukyl/rt-detrv2-heron-101-rukopys"
model = AutoModelForObjectDetection.from_pretrained(repo).eval()
processor = AutoImageProcessor.from_pretrained(repo)

image = Image.open("page.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# threshold low — the scores are diffuse; max-F1 is at ~0.075 (see Limitations)
target_sizes = torch.tensor([image.size[::-1]])  # (height, width)
result = processor.post_process_object_detection(
    outputs, threshold=0.1, target_sizes=target_sizes
)[0]
for score, label, box in zip(result["scores"], result["labels"], result["boxes"]):
    print(model.config.id2label[label.item()], round(score.item(), 3), box.tolist())
```

## Files

```
model.safetensors, config.json,   the shipped checkpoint — loads directly
preprocessor_config.json          with from_pretrained
training_meta.json                recorded recipe + the shipped checkpoint's metrics
metrics.json                      val metrics, aggregate + per-class
selection_comparison.json         the three checkpoint-selection axes side by side
training_log.jsonl                per-epoch loss / mAP / P / R curves
```

## Head re-initialisation (17 → 7)

The backbone, encoder, decoder, and the class-agnostic box-regression head load
from the base checkpoint as-is; the classification head is re-initialised to the
7 Rukopys classes, with `id2label`/`label2id` baked into the config.

## Training

Fine-tuned on the human-labeled Rukopys train split via the 🤗 `Trainer`. The
image processor applies a deterministic 640×640 resize with ImageNet
normalisation.

### Hyperparameters (as launched)

| hyperparameter | value |
|---|---|
| epochs | 60 (early stopping, patience 15 on val loss) |
| batch / image size | 32 / 640 |
| optimizer / schedule | AdamW, linear decay |
| learning rate | 1e-4 |
| warmup ratio | 0.1 |
| weight decay | 1e-4 |
| freeze | none |
| augmentation | none |
| eval / checkpoint cadence | every epoch |
| checkpoint | best val mAP@50 (epoch 20) |
| seed | 42 |

The full recorded recipe ships in `training_meta.json`; per-epoch curves in
`training_log.jsonl`.

### Checkpoint selection

The run tracked three selection axes; val loss and val mAP@50 are anti-correlated
across this run (Pearson ≈ −0.78). This repo ships the mAP@50-selected
checkpoint.

| selected on | mAP@50 | mAP@50-95 | precision | recall |
|---|---:|---:|---:|---:|
| val loss | 0.7444 | 0.4731 | 0.9583 | 0.0175 |
| val mAP@50 (shipped) | 0.7685 | 0.4819 | 0.9730 | 0.0137 |
| val recall | 0.1055 | 0.0590 | 0.7929 | 0.4690 |

### Class distribution (region counts)

| class | train | val |
|---|---:|---:|
| handwritten | 19,420 | 2,157 |
| printed | 266 | 42 |
| formula | 2,545 | 347 |
| table | 128 | 14 |
| annotation | 494 | 60 |
| image | 115 | 4 |
| graph | 54 | 5 |
| total | 23,022 | 2,629 |

## Results

Held-out val split of the Rukopys train data: 133 pages / 2,629 regions
(stratified holdout, seed 42). mAP is threshold-free with box matching at IoU 0.50
(mAP@50–95 averages IoU 0.50:0.95); precision and recall are reported at a fixed
confidence ≥ 0.50 operating point (class-aware greedy matching at IoU ≥ 0.50).
The model's per-class sigmoid scores are diffuse and few boxes clear 0.50, so
recall at this operating point collapses to ~0.01 — an artifact of the threshold,
not the miss rate; the threshold-free mAP is the signal. Per-class
precision/recall are not recorded.

### Aggregate (macro over 7 classes)

| metric | value |
|---|---:|
| mAP@50 | 0.7685 |
| mAP@50-95 | 0.4819 |
| precision | 0.9730 |
| recall | 0.0137 |

### Per class

| class | n | mAP@50 | mAP@50-95 |
|---|---:|---:|---:|
| handwritten | 2,157 | 0.8478 | 0.5042 |
| printed | 42 | 0.8213 | 0.4453 |
| formula | 347 | 0.7973 | 0.5242 |
| table | 14 | 0.8553 | 0.4179 |
| annotation | 60 | 0.3246 | 0.1567 |
| image | 4 | 0.9505 | 0.7693 |
| graph | 5 | 0.7830 | 0.5555 |

We also acknowledge that `printed`/`table`/`annotation`/`image`/`graph` n is quite
small, so measuring detection metrics against them is quite noisy.

## Limitations

- Confidence is uncalibrated: the empirical max-F1 confidence is ≈0.075, far
  below a typical 0.25–0.50 first guess, and F1 collapses by conf 0.30. Tune the
  inference threshold against your own target metric.
- `annotation` (mAP@50 0.32) is a weak class.
- Trained and evaluated at 640×640; very small or dense regions on
  high-resolution scans may benefit from a higher inference resolution.
- Handwritten Ukrainian school/archival-style pages only; behaviour on other
  document types is untested.
- Single seed and validation split — no across-run variance estimate.

## Training data & attribution

| dataset | source | license | role |
|---|---|---|---|
| Rukopys | [`UkrainianCatholicUniversity/rukopys`](https://huggingface.co/datasets/UkrainianCatholicUniversity/rukopys) | CC BY 4.0 | gold fine-tune |

Model weights are Apache-2.0, inherited from RT-DETRv2 and the base checkpoint.