chess-guesser-distilbert

This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.6731
• Accuracy: 0.585

Disclaimer

This model is usable, but not very accurate at higher levels of games. This model is geared towards elo below 1200. This model is my final project for college. Any recommendation or advice is greatly appreciated.

Model description

This model is a fine-tuned Binary Classifier based on the distilbert-base-uncased architecture. It is specifically designed to analyze the "Language of Chess" by processing PGN (Portable Game Notation) move sequences. The model distinguishes between two skill-based categories:

Bucket 01: Novice (400–1000 Elo)

Bucket 02: Intermediate (1001–1200 Elo)

Unlike traditional engines like Stockfish that calculate mathematical optimality, this model uses Self-Attention to identify stylistic patterns and tactical consistency common in human play.

Intended uses & limitations

Intended Uses:

Educational Elo Estimation: Providing casual players with a skill-level estimate based on their actual game moves rather than just their win/loss ratio.

Fair-Play Screening: Assisting tournament administrators, specifically for organizations like the Wisconsin Scholastic Chess Federation, in identifying performances that significantly exceed a player's registered rating.

Limitations:

Skill Range: The model is currently limited to the 400–1200 Elo range. It is not designed to analyze Master-level or Grandmaster-level games.

PGN Length: Best results are achieved with at least 15–20 moves; very short games may not provide enough "linguistic" context for an accurate classification.

Training and evaluation data

The model was trained on a balanced subset of 2,000 games derived from the Lichess Open Database (2025).

Preprocessing: Critical cleaning was performed to ensure zero data leakage. All PGN metadata headers (including [WhiteElo] and [BlackElo]) were stripped using a custom Python parser, forcing the model to rely strictly on move notation.

Formatting: Move strings were cleaned of newlines and extra spacing to create a continuous sequence for the DistilBERT tokenizer.

Training Procedure

Preprocessing Strategy The training pipeline utilized a custom-built "Header-Stripper" to ensure the model learned chess strategy rather than metadata.

Data Cleaning: Regex patterns were used to identify the end of the PGN header block and extract only the move notation beginning with 1..

Tokenization: I used the DistilBertTokenizer with a max_length of 512. This captures the opening and early middlegame, which are high-signal areas for distinguishing player skill.

Normalization: All moves were flattened into a single-line string, removing newlines and comments to keep the input sequence "clean" for the transformer's attention layers.

Training Hardware & Environment Hardware: The model was trained using an NVIDIA T4 GPU provided via Google Colab.

Mixed Precision: I enabled fp16 (Floating Point 16) training to reduce the memory footprint and speed up the calculation of the classification head.

Runtime: The total training time was approximately 184.67 seconds, demonstrating the efficiency of the DistilBERT architecture for rapid MVP development.

Hyperparameters and Optimization Optimizer: AdamW with a weight decay of 0.01 was used to prevent the model from overfitting on specific chess openings.

Learning Rate: Set at 2e-5, a standard "fine-tuning" rate that allows the model to adapt to chess notation without losing the general linguistic context learned from the base DistilBERT model.

Batch Size: 16 for training and 8 for evaluation.

Epochs: 3 total passes through the dataset.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.6766	1.0	113	0.6805	0.59
0.6832	2.0	226	0.6731	0.585
0.6414	3.0	339	0.6736	0.575

Framework versions

• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2