Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- sw
|
| 4 |
+
tags:
|
| 5 |
+
- sign-language
|
| 6 |
+
- pose-landmarks
|
| 7 |
+
- mediapipe
|
| 8 |
+
- transformer
|
| 9 |
+
- african-sign-language
|
| 10 |
+
- KSL
|
| 11 |
+
- isolated-sign-recognition
|
| 12 |
+
license: apache-2.0
|
| 13 |
+
datasets:
|
| 14 |
+
- luciayen/KSL-Hand-Landmarks
|
| 15 |
+
metrics:
|
| 16 |
+
- accuracy
|
| 17 |
+
model-index:
|
| 18 |
+
- name: AfriSignEncoder Exp1 — KSL Baseline
|
| 19 |
+
results:
|
| 20 |
+
- task:
|
| 21 |
+
type: video-classification
|
| 22 |
+
name: Isolated Sign Word Recognition
|
| 23 |
+
dataset:
|
| 24 |
+
name: KSL-Hand-Landmarks
|
| 25 |
+
type: luciayen/KSL-Hand-Landmarks
|
| 26 |
+
metrics:
|
| 27 |
+
- type: accuracy
|
| 28 |
+
value: 1.0
|
| 29 |
+
name: Validation Accuracy
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
# AfriSignEncoder Exp1 — KSL Baseline (LandmarkTransformer)
|
| 33 |
+
|
| 34 |
+
Part of the **AfriSignEncoder** research project: a multilingual African sign language recognition benchmark.
|
| 35 |
+
This checkpoint is the **Experiment 1 single-language baseline** for Kenyan Sign Language (KSL).
|
| 36 |
+
|
| 37 |
+
> **Scope note:** This baseline covers only 4 KSL glosses (father, hello, is, my) from a small
|
| 38 |
+
> demo-scale dataset. The 100% accuracy reflects the simplicity of the 4-class task and should
|
| 39 |
+
> not be generalised to full KSL vocabulary recognition.
|
| 40 |
+
|
| 41 |
+
## Model Description
|
| 42 |
+
|
| 43 |
+
Same LandmarkTransformer architecture as the CASL baseline, with a 4-class head.
|
| 44 |
+
|
| 45 |
+
| Component | Value |
|
| 46 |
+
|---|---|
|
| 47 |
+
| Architecture | LandmarkTransformer (custom) |
|
| 48 |
+
| Input | (B, 64, 225) float32 — 75 keypoints × 3 coords per frame |
|
| 49 |
+
| Embedding dim | 256 |
|
| 50 |
+
| Attention heads | 8 |
|
| 51 |
+
| Encoder layers | 4 |
|
| 52 |
+
| Feed-forward dim | 1,024 |
|
| 53 |
+
| Positional encoding | Learned |
|
| 54 |
+
| Classification head | Linear 256 → 4 |
|
| 55 |
+
| Parameters | ~3.24 M |
|
| 56 |
+
|
| 57 |
+
## Dataset
|
| 58 |
+
|
| 59 |
+
**KSL Hand Landmarks** — 4 Kenyan Sign Language words.
|
| 60 |
+
|
| 61 |
+
| Split | Samples | Classes |
|
| 62 |
+
|---|---|---|
|
| 63 |
+
| Train | 694 | 4 |
|
| 64 |
+
| Test | 124 | 4 |
|
| 65 |
+
|
| 66 |
+
Source: Kaggle `joanwachuka/ksl-hand-landmarks` → parquet at `luciayen/KSL-Hand-Landmarks`.
|
| 67 |
+
|
| 68 |
+
**Landmark caveat:** The original `.npy` files contain MediaPipe **Hands**-only keypoints
|
| 69 |
+
(42 joints × 3 = 126D). These are placed in dimensions [0:126]; dimensions [126:225]
|
| 70 |
+
(the 99 pose body dims) are zero-padded. The model therefore learns from hand shape
|
| 71 |
+
and motion only; the padded dimensions are always zero and contribute no signal.
|
| 72 |
+
|
| 73 |
+
**Data leakage fix:** The Kaggle archive contains two sub-directories — `data_split/`
|
| 74 |
+
(official train/val/test) and `dataset3/` (pre-split source). All 124 test samples appear
|
| 75 |
+
verbatim in `dataset3/`. The upload script excludes `dataset3/` entirely, giving a clean
|
| 76 |
+
train=694 / test=124 split with zero overlap.
|
| 77 |
+
|
| 78 |
+
## Training
|
| 79 |
+
|
| 80 |
+
Identical protocol to the CASL baseline.
|
| 81 |
+
|
| 82 |
+
| Setting | Value |
|
| 83 |
+
|---|---|
|
| 84 |
+
| Optimiser | AdamW (lr=3e-4, wd=1e-4) |
|
| 85 |
+
| LR schedule | OneCycleLR cosine |
|
| 86 |
+
| Max epochs | 60 |
|
| 87 |
+
| Batch size | 64 |
|
| 88 |
+
| Loss | CrossEntropy + label_smoothing=0.1 |
|
| 89 |
+
| Early stopping | patience=12 on val acc |
|
| 90 |
+
| Normalisation | Per-feature z-score (stats stored in checkpoint) |
|
| 91 |
+
|
| 92 |
+
## Results
|
| 93 |
+
|
| 94 |
+
| Metric | Value |
|
| 95 |
+
|---|---|
|
| 96 |
+
| Best validation accuracy | **100%** |
|
| 97 |
+
| Best checkpoint epoch | 3 |
|
| 98 |
+
| Final epoch (early stop) | 15 |
|
| 99 |
+
| Number of classes | 4 |
|
| 100 |
+
|
| 101 |
+
The 100% result is expected: 4 highly phonologically distinct signs, ~173 training samples
|
| 102 |
+
per class, and a well-regularised model. This result validates the pipeline; it does not
|
| 103 |
+
benchmark KSL at meaningful scale.
|
| 104 |
+
|
| 105 |
+
## Checkpoint Contents
|
| 106 |
+
|
| 107 |
+
```python
|
| 108 |
+
import torch
|
| 109 |
+
ck = torch.load("pytorch_model.bin", map_location="cpu")
|
| 110 |
+
# Keys: epoch, val_acc, model (state_dict), l2i (label→index dict),
|
| 111 |
+
# mean (tensor 225,), std (tensor 225,)
|
| 112 |
+
# l2i = {"father": 0, "hello": 1, "is": 2, "my": 3}
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
## Limitations
|
| 116 |
+
|
| 117 |
+
- Only 4 classes — not a meaningful KSL benchmark.
|
| 118 |
+
- Pose dims are always zero (hand-only source data). Re-extraction with MediaPipe Holistic is planned.
|
| 119 |
+
- Small dataset; results will change with more data.
|
| 120 |
+
|
| 121 |
+
## Citation / Project
|
| 122 |
+
|
| 123 |
+
AfriSignEncoder research project, CMU, 2026. GitHub: `africansl_encoder`.
|