ASL Citizen Small Transformer Isolated Sign Encoder
This repository contains the trained Small Transformer Encoder model for isolated American Sign Language recognition.
The model was trained as a comparison baseline for the FYP project:
Bidirectional ASL ↔ English Translation System
The goal of this experiment was to compare a self-attention-based temporal encoder against the previously trained BiGRU + Attention baseline.
Project Stage
ASL Citizen processed keypoints-200
↓
Small Transformer isolated sign recognition model
↓
Comparison with BiGRU + Attention baseline
↓
Select stronger encoder for future How2Sign CTC stage
Dataset
Training dataset:
SharoonArshad/asl-citizen-processed-200
The dataset uses the processed ASL Citizen keypoints-200 format:
(N, 200, 450)
Where:
200 = fixed sequence length
450 = 225 position features + 225 velocity features
225 = 75 landmarks × 3 coordinates
75 landmarks = 33 pose + 21 left hand + 21 right hand
Model Architecture
The model uses a small Transformer encoder over temporal keypoint sequences:
Input: (B, 200, 450)
Input LayerNorm: 450
Linear Projection: 450 → 256
CLS Token: prepended to the sequence
Positional Encoding: sinusoidal
Transformer Encoder: 3 layers
Attention Heads: 4
Feedforward Dimension: 768
Pooling: CLS token
Classifier Head: 200 ASL classes
Training Configuration
Main training settings:
Batch size: 32
Max epochs: 60
Best epoch: 52
Early stopping patience: 10
Learning rate: 0.0002
Weight decay: 0.0001
Dropout: 0.3
Label smoothing: 0.1
Gradient clipping: 1.0
Scheduler: cosine
Optimizer: AdamW
Mixed precision: enabled on GPU
Results
Small Transformer final results:
Best epoch: 52
Best validation Top-1 accuracy: 78.13%
Test loss: 1.3240
Test correct: 3238/3552
Test Top-1 accuracy: 91.16%
Test Top-5 accuracy: 99.13%
Test Macro F1: 0.9118
Test Micro F1: 0.9119
Test Weighted F1: 0.9104
Mean per-class accuracy: 0.9144
Training time: 06m 40s
Comparison with BiGRU + Attention Baseline
The BiGRU + Attention model achieved stronger performance on the same ASL Citizen processed dataset:
Metric BiGRU + Attention Small Transformer
Validation Top-1 85.47% 78.13%
Test Top-1 98.48% 91.16%
Test Top-5 99.58% 99.13%
Test Macro F1 0.9850 0.9118
Test Weighted F1 0.9848 0.9104
Training Time 26m 25s 06m 40s
Interpretation
The Small Transformer trained successfully, but it underperformed the BiGRU + Attention model.
The best validation Top-1 accuracy was:
BiGRU + Attention: 85.47%
Small Transformer: 78.13%
Therefore, the BiGRU + Attention encoder was selected as the stronger isolated sign encoder for future transfer into the continuous How2Sign CTC stage.
The Small Transformer is still useful as a comparison baseline for the thesis because it demonstrates that a self-attention temporal encoder was evaluated against a recurrent attention-based encoder.
Repository Files
asl_isolated_small_transformer.zip
checkpoints/best_small_transformer_model.pt
checkpoints/best_transformer_encoder_only.pt
checkpoints/last_small_transformer_model.pt
reports/final_metrics.json
reports/training_history.csv
reports/test_classification_report.csv
reports/test_classification_report.json
reports/test_confusion_matrix.npy
reports/test_per_class_accuracy.csv
metadata/label_to_id.json
metadata/id_to_label.json
training_config.json
Important Files
Full model checkpoint
checkpoints/best_small_transformer_model.pt
Use this file if you want to reload the complete Small Transformer classifier for isolated sign prediction.
Encoder-only checkpoint
checkpoints/best_transformer_encoder_only.pt
Use this file if you want to reuse the Transformer encoder in future experiments.
Full ZIP archive
asl_isolated_small_transformer.zip
This archive contains the full training output folder, including checkpoints, reports, label maps, and configuration files.
Loading the Model in Kaggle
Example download code:
from huggingface_hub import hf_hub_download
repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"
model_path = hf_hub_download(
repo_id=repo_id,
repo_type="model",
filename="checkpoints/best_small_transformer_model.pt"
)
encoder_path = hf_hub_download(
repo_id=repo_id,
repo_type="model",
filename="checkpoints/best_transformer_encoder_only.pt"
)
print(model_path)
print(encoder_path)
Loading the Full ZIP Archive
from huggingface_hub import hf_hub_download
import zipfile
from pathlib import Path
repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"
zip_path = hf_hub_download(
repo_id=repo_id,
repo_type="model",
filename="asl_isolated_small_transformer.zip"
)
extract_dir = Path("/kaggle/working/asl_isolated_small_transformer_loaded")
extract_dir.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(zip_path, "r") as zip_ref:
zip_ref.extractall(extract_dir)
print("Extracted to:", extract_dir)
Thesis Use
This model can be documented as the Transformer-based isolated sign recognition baseline.
Suggested thesis statement:
A Small Transformer Encoder was trained on the processed ASL Citizen keypoints-200 dataset for 200 isolated ASL classes. The model achieved 78.13% validation Top-1 accuracy and 91.16% test Top-1 accuracy. Although it trained faster than the BiGRU + Attention model, it achieved lower validation and test performance. Therefore, the BiGRU + Attention encoder was selected as the primary isolated sign encoder for future CTC-based continuous sign recognition.
Limitations
- The model is trained on isolated signs, not continuous signing.
- It recognizes 200 selected ASL classes from the processed ASL Citizen dataset.
- It underperformed the BiGRU + Attention baseline on validation accuracy and test accuracy.
- It does not yet perform ASL sentence translation.
- A larger Transformer may require more data or stronger regularization.
Next Step
The next project stage is to use the stronger BiGRU + Attention encoder checkpoint as the pretrained visual encoder for a CTC-based How2Sign continuous sign recognition model.