---
license: apache-2.0
library_name: coreml
tags:
  - coreml
  - fingerspelling
  - sign-language
  - ctc
---

# Fingerspelling CTC (Core ML)

On-device American Sign Language **fingerspelling** recognizer used by the OpenGlasses
accessibility tier. A CTC sequence model over MediaPipe Holistic landmarks, converted to
a fixed-window Core ML program:

- **Input**: `features` `(1, 768, 1629)` float32 — up to 768 frames × 543 landmarks × xyz
  (canonical order: face 0–467, left hand 468–488, pose 489–521, right hand 522–542),
  per-window standardised, NaN→0, zero-padded — plus `mask` `(1, 768)` float32 validity.
- **Output**: `(1, 384, 62)` CTC logits (stride 2; read the first ⌈T/2⌉ rows; class 0 is
  the blank, classes 1–59 map through `vocab.txt`, the last two are auxiliary tokens).
- Greedy decode scores **20.8% mean CER (median 10.8%)** on a 300-sequence held-out set
  of the training corpus's rerun benchmark.

## Files

| File | Purpose |
|---|---|
| `Fingerspelling2P.mlpackage` | the Core ML model (fp16, iOS 17+) |
| `vocab.txt` | CTC charset sidecar (`<blank>` + 59 symbols, one per line) |
| `holistic_landmarker.task` | MediaPipe holistic landmark extractor consumed alongside the model |

## License & attribution

- Model weights and conversion: **Apache-2.0** (see `LICENSE`).
- Trained on the Google ASL Fingerspelling corpus (Google / Deaf Professional Arts
  Network), **CC-BY 4.0**.
- `holistic_landmarker.task` is a MediaPipe model asset (Google), **Apache-2.0**.