pszemraj/xtremedistil-l6-h256-OCR-quality

xtremedistil-l6-h256-uncased: OCR-quality-classification

This model is a fine-tuned version of microsoft/xtremedistil-l6-h256-uncased on pszemraj/OCR-quality-classification

It achieves the following results on the evaluation set:

• Loss: 0.0316
• Accuracy: 0.994
• Num Input Tokens Seen: 57341952

Intended uses & limitations

predict whether a document is clean or noisy

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Input Tokens Seen
0.0812	0.2660	250	0.0860	0.986	8192000
0.0637	0.5321	500	0.0532	0.988	16384000
0.031	0.7981	750	0.0463	0.99	24576000
0.0315	1.0641	1000	0.0343	0.992	32765952
0.0223	1.3301	1250	0.0337	0.994	40957952
0.0137	1.5962	1500	0.0423	0.99	49149952
0.0186	1.8622	1750	0.0316	0.994	57341952

Framework versions

• Transformers 4.40.2
• Pytorch 2.2.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1