rt-detrv2-heron-101-rukopys

An RT-DETRv2-R101 (RTDetrV2ForObjectDetection, ~76.7M params) document-layout detector for handwritten Ukrainian pages, fine-tuned on the Rukopys dataset with a 7-class head: 0 handwritten · 1 printed · 2 formula · 3 table · 4 annotation · 5 image · 6 graph.

TL;DR

	value
Architecture	RT-DETRv2, ResNet-101 backbone, RTDetrV2ForObjectDetection
Parameters	~76.7M
Init from	docling-project/docling-layout-heron-101 (17-class document-layout checkpoint)
Classes	handwritten, printed, formula, table, annotation, image, graph
Decoding	set prediction, 300 object queries, no NMS
Gold-val mAP@50 / mAP@50-95	0.7685 / 0.4819 (macro over 7 classes)
Input	a document page image, 640×640
Output	class-labeled region bounding boxes

Intended use

Detecting and classifying page regions on handwritten Ukrainian documents, upstream of region recognizers (e.g. Hukyl/trocr-large-rukopys for text, Hukyl/trocr-base-rukopys-formula for formulas). For a stronger single detector on the same task, see Hukyl/doclayout-yolov10-rukopys.

How to use

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

repo = "Hukyl/rt-detrv2-heron-101-rukopys"
model = AutoModelForObjectDetection.from_pretrained(repo).eval()
processor = AutoImageProcessor.from_pretrained(repo)

image = Image.open("page.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# threshold low — the scores are diffuse; max-F1 is at ~0.075 (see Limitations)
target_sizes = torch.tensor([image.size[::-1]])  # (height, width)
result = processor.post_process_object_detection(
    outputs, threshold=0.1, target_sizes=target_sizes
)[0]
for score, label, box in zip(result["scores"], result["labels"], result["boxes"]):
    print(model.config.id2label[label.item()], round(score.item(), 3), box.tolist())

Files

model.safetensors, config.json,   the shipped checkpoint — loads directly
preprocessor_config.json          with from_pretrained
training_meta.json                recorded recipe + the shipped checkpoint's metrics
metrics.json                      val metrics, aggregate + per-class
selection_comparison.json         the three checkpoint-selection axes side by side
training_log.jsonl                per-epoch loss / mAP / P / R curves

Head re-initialisation (17 → 7)

The backbone, encoder, decoder, and the class-agnostic box-regression head load from the base checkpoint as-is; the classification head is re-initialised to the 7 Rukopys classes, with id2label/label2id baked into the config.

Training

Fine-tuned on the human-labeled Rukopys train split via the 🤗 Trainer. The image processor applies a deterministic 640×640 resize with ImageNet normalisation.

Hyperparameters (as launched)

hyperparameter	value
epochs	60 (early stopping, patience 15 on val loss)
batch / image size	32 / 640
optimizer / schedule	AdamW, linear decay
learning rate	1e-4
warmup ratio	0.1
weight decay	1e-4
freeze	none
augmentation	none
eval / checkpoint cadence	every epoch
checkpoint	best val mAP@50 (epoch 20)
seed	42

The full recorded recipe ships in training_meta.json; per-epoch curves in training_log.jsonl.

Checkpoint selection

The run tracked three selection axes; val loss and val mAP@50 are anti-correlated across this run (Pearson ≈ −0.78). This repo ships the mAP@50-selected checkpoint.

selected on	mAP@50	mAP@50-95	precision	recall
val loss	0.7444	0.4731	0.9583	0.0175
val mAP@50 (shipped)	0.7685	0.4819	0.9730	0.0137
val recall	0.1055	0.0590	0.7929	0.4690

Class distribution (region counts)

class	train	val
handwritten	19,420	2,157
printed	266	42
formula	2,545	347
table	128	14
annotation	494	60
image	115	4
graph	54	5
total	23,022	2,629

Results

Held-out val split of the Rukopys train data: 133 pages / 2,629 regions (stratified holdout, seed 42). mAP is threshold-free with box matching at IoU 0.50 (mAP@50–95 averages IoU 0.50:0.95); precision and recall are reported at a fixed confidence ≥ 0.50 operating point (class-aware greedy matching at IoU ≥ 0.50). The model's per-class sigmoid scores are diffuse and few boxes clear 0.50, so recall at this operating point collapses to ~0.01 — an artifact of the threshold, not the miss rate; the threshold-free mAP is the signal. Per-class precision/recall are not recorded.

Aggregate (macro over 7 classes)

metric	value
mAP@50	0.7685
mAP@50-95	0.4819
precision	0.9730
recall	0.0137

Per class

class	n	mAP@50	mAP@50-95
handwritten	2,157	0.8478	0.5042
printed	42	0.8213	0.4453
formula	347	0.7973	0.5242
table	14	0.8553	0.4179
annotation	60	0.3246	0.1567
image	4	0.9505	0.7693
graph	5	0.7830	0.5555

We also acknowledge that printed/table/annotation/image/graph n is quite small, so measuring detection metrics against them is quite noisy.

Limitations

• Confidence is uncalibrated: the empirical max-F1 confidence is ≈0.075, far below a typical 0.25–0.50 first guess, and F1 collapses by conf 0.30. Tune the inference threshold against your own target metric.
• annotation (mAP@50 0.32) is a weak class.
• Trained and evaluated at 640×640; very small or dense regions on high-resolution scans may benefit from a higher inference resolution.
• Handwritten Ukrainian school/archival-style pages only; behaviour on other document types is untested.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

dataset	source	license	role
Rukopys	UkrainianCatholicUniversity/rukopys	CC BY 4.0	gold fine-tune

Model weights are Apache-2.0, inherited from RT-DETRv2 and the base checkpoint.