| --- |
| language: |
| - bo |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - image-classification |
| - dinov3 |
| - tibetan |
| - script-classification |
| - paleography |
| - fine-tuned |
| - document-analysis |
| base_model: facebook/dinov3-vits16-pretrain-lvd1689m |
| datasets: |
| - custom |
| metrics: |
| - f1 |
| - accuracy |
| pipeline_tag: image-classification |
| model-index: |
| - name: Tibetan Script Classifier (DINOv3 ViT-S) |
| results: |
| - task: |
| type: image-classification |
| name: Tibetan Script Classification |
| metrics: |
| - name: Macro F1 (whole page) |
| type: f1 |
| value: 0.512 |
| - name: Accuracy (whole page) |
| type: accuracy |
| value: 0.571 |
| - name: Macro F1 (CLAHE patches, page-level) |
| type: f1 |
| value: 0.529 |
| --- |
| |
| # Tibetan Script Classifier (DINOv3) |
|
|
| This repository contains fine-tuned Tibetan script classification checkpoints for 18 classes, trained from the DINOv3 ViT-S backbone: |
|
|
| - Backbone: `facebook/dinov3-vits16-pretrain-lvd1689m` |
| - Task: 18-way script classification |
| - Training script included: `finetune_dinov3.py` |
|
|
| **Hugging Face access:** DINOv3 requires access approval at [huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m) before `from_pretrained` / downloads will work. Anyone cloning this repo will see the same gated-model error until their HF account is granted access and they are logged in (`huggingface-cli login` or `HF_TOKEN`). |
|
|
| ## Label Set |
|
|
| `dhumri`, `difficult`, `drathung`, `drudring`, `druring`, `druthung`, `khyuyig`, `multi_scripts`, `non_tibetan`, `peri`, `petsuk`, `trinyig`, `tsegdrig`, `tsugchung`, `tsumachug`, `uchen_sugdring`, `uchen_sugthung`, `yigchung`. |
|
|
| ## Preprocessing (per experiment) |
|
|
| Images for training were produced as follows (see `preprocess.py` in the parent project): |
|
|
| - **`whole_page`:** resize so the **short edge is 224 px**, then **center crop** to 224×224 (one crop per source page). |
| - **`patches_color`:** same short-edge resize to 224, then **sliding-window** 224×224 patches with **25% overlap** between windows (multiple crops per page). |
| - **`patches_clahe`:** identical patch layout as `patches_color`; each patch is converted to grayscale and **CLAHE** contrast normalization is applied (`clipLimit=2.0`, `tileGridSize=(8,8)`), then saved as BGR/RGB for training. |
|
|
| ## Training recipe |
|
|
| - **Progressive unfreezing (defaults in `finetune_dinov3.py`):** |
| - **Stage A — head only:** 20 epochs, backbone frozen, classifier head LR **1e-3** (backbone LR 0). |
| - **Stage B — last 2 blocks:** 10 epochs, backbone LR **1e-5**, head LR **1e-3**. |
| - **Stage C — last 4 blocks:** 10 epochs, backbone LR **5e-6**, head LR **5e-4**. |
| - **Loss:** class-weighted **cross-entropy** with inverse-frequency weights over the training split (`nn.CrossEntropyLoss(weight=...)`). |
| - **Sampling:** the published runs use a standard `DataLoader` with **`shuffle=True`**. The script also defines **`get_weighted_sampler` → `WeightedRandomSampler`** if you want to switch the train loader to explicit class-balanced sampling. |
| - **Document-aware augmentations (train only):** `RandomRotation` **±5°** (fill white), `ColorJitter` brightness/contrast **±20%** (`0.2`), plus `RandomResizedCrop` and light `RandomErasing` as in `ScriptDataset`; **no horizontal flip**. |
| |
| ## Class distribution (`whole_page` split totals) |
| |
| The whole-page split has **5,684** samples in total (**train 3,996 / val 844 / test 844**). The **844** figure in test metrics is **only the test split** (roughly 15% of pages per class held out for testing), not the full dataset. |
|
|
| The per-class table below sums **train + val + test** counts. A benchmark exclusion manifest (`benchmark_page_ids.json`, **88** page IDs across 18 classes) is consulted when building splits: any file whose page ID matches is skipped and counted in `splits.json` under `skipped_excluded_files_by_class`. For the published `whole_page/splits.json`, that skip map is **empty**—so either those pages were **not present** under the training `data_dir`, or they had **already been removed** before this split was built. The **5,684** totals are whatever files remained for stratified splitting. |
|
|
| | Class | Samples | |
| |---|---:| |
| | dhumri | 98 | |
| | difficult | 170 | |
| | drathung | 129 | |
| | drudring | 132 | |
| | druring | 119 | |
| | druthung | 207 | |
| | khyuyig | 113 | |
| | multi_scripts | 235 | |
| | non_tibetan | 192 | |
| | peri | 614 | |
| | petsuk | 1388 | |
| | trinyig | 42 | |
| | tsegdrig | 749 | |
| | tsugchung | 77 | |
| | tsumachug | 178 | |
| | uchen_sugdring | 835 | |
| | uchen_sugthung | 240 | |
| | yigchung | 166 | |
|
|
| ## Experiments Included |
|
|
| ### 1) `whole_page` |
| |
| - Files: `whole_page/final_model.pt`, `results.json`, `confusion_matrix.png`, `confusion_matrix.csv`, `splits.json` |
| - Test (image-level) macro-F1: **0.5124** |
| - Test accuracy: **0.5711** |
| |
| ### 2) `patches_color` |
|
|
| - Files: `patches_color/final_model.pt`, `results.json`, `confusion_matrix.png`, `checkpoint_page_eval.json`, `splits.json` |
| - Test (patch-level) macro-F1: **0.4899** |
| - Re-eval **page-level** macro-F1 for shipped `final_model.pt` (`checkpoint_page_eval.json`): **0.5017** |
| - Best **page-level** macro-F1 among stage checkpoints on the same grid: **0.5043** (**Stage A**) |
|
|
| ### 3) `patches_clahe` |
| |
| - Files: `patches_clahe/final_model.pt`, `results.json`, `confusion_matrix.png`, `checkpoint_page_eval.json`, `splits.json` |
| - Test (patch-level) macro-F1: **0.4911** |
| - Re-eval page-level macro-F1 for shipped `final_model.pt`: **0.5261** |
| - Best **page-level** macro-F1 among stage checkpoints: **0.529** (**Stage B**) |
|
|
| ## Which stage produced which checkpoint? |
|
|
| - **`final_model.pt` in each folder** is the stage with the highest **validation macro-F1** among `best_stage_*.pt` checkpoints (see `best_val_checkpoint` in each `results.json`): **Stage B** for `whole_page`, **Stage C** for both `patches_color` and `patches_clahe`. |
| - For **page-level** quality on the patch runs, the best single stage on the re-eval grid differs: **Stage A** (`patches_color`) and **Stage B** (`patches_clahe`) beat their respective `final_model.pt` page scores—use `checkpoint_page_eval.json` if you want to deploy a stage checkpoint instead of the val-selected default. |
|
|
| ## Which experiment won? |
|
|
| CLAHE patches achieved the highest **page-level** macro-F1 (**0.529** on the best stage checkpoint), while **whole page** achieved the best **image-level** macro-F1 (**0.512**). **Whole page** is recommended for production due to simpler inference. |
|
|
| ## How To Load a Checkpoint |
|
|
| ```python |
| import torch |
| from pathlib import Path |
| from finetune_dinov3 import DINOv3Classifier, DINOV3_MODEL_ID |
| |
| ckpt_path = Path("whole_page/final_model.pt") |
| payload = torch.load(ckpt_path, map_location="cpu") |
| |
| label_to_idx = payload["label_to_idx"] |
| idx_to_label = {v: k for k, v in label_to_idx.items()} |
| num_classes = len(label_to_idx) |
| |
| model = DINOv3Classifier(DINOV3_MODEL_ID, num_classes) |
| model.load_state_dict(payload["model_state_dict"]) |
| model.eval() |
| ``` |
|
|
| ## Inference (Single Image) |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from transformers import AutoImageProcessor |
| |
| processor = AutoImageProcessor.from_pretrained(DINOV3_MODEL_ID) |
| img = Image.open("example.png").convert("RGB") |
| inputs = processor(images=img, return_tensors="pt") |
| |
| with torch.no_grad(): |
| logits = model(inputs["pixel_values"]) |
| probs = torch.softmax(logits, dim=1)[0].cpu().numpy() |
| pred_idx = int(probs.argmax()) |
| pred_label = idx_to_label[pred_idx] |
| print(pred_label, float(probs[pred_idx])) |
| ``` |
|
|
| ## Page-Level Inference (Patch Aggregation) |
|
|
| For patch experiments (`patches_color`, `patches_clahe`), aggregate by page stem: |
|
|
| 1. group patch probabilities by page ID (strip `_pN` suffix), |
| 2. average probabilities per page, |
| 3. take `argmax` of averaged probabilities. |
|
|
| This is the same logic used in the re-evaluation script output (`checkpoint_page_eval.json`). |
|
|
| ## Known Limitations |
|
|
| - Class imbalance is high (for example `petsuk` and `uchen_sugdring` dominate, while `trinyig` is small). |
| - Results can vary by preprocessing variant and by patch vs page-level evaluation protocol. |
| - Patch-level metrics and page-level metrics are not directly interchangeable. |
| - The model expects Tibetan manuscript-style inputs; performance can drop on out-of-domain scans or mixed/noisy pages. |
| - Checkpoints are tied to the exact label mapping saved in each payload (`label_to_idx`). |
|
|
| ## Reproducibility Notes |
|
|
| - Exclusion manifest support is enabled in training (`benchmark_page_ids.json`). |
| - Full training code used for these artifacts is included at `finetune_dinov3.py`. |
|
|
