Daryl Lim Claude Opus 4.6 commited on
Commit
d999e21
·
1 Parent(s): 491ad42

Update docs for evaluated-only language set

Browse files

Update CLAUDE.md and README.md to reflect 183 evaluated languages
from Table 14, Section A.9.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (2) hide show
  1. CLAUDE.md +2 -2
  2. README.md +2 -2
CLAUDE.md CHANGED
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
4
 
5
  ## Project Overview
6
 
7
- A Hugging Face Spaces app that translates English text to ~400 languages using Google's [MADLAD-400](https://arxiv.org/pdf/2309.04662) 3B Seq2Seq model. Built with Gradio and deployed on HF Spaces. Falls back to CPU with a warning when no GPU is available.
8
 
9
  ## Commands
10
 
@@ -34,7 +34,7 @@ pytest -m slow # model tests only
34
 
35
  **`app.py`** — Single-file application. Uses `@lru_cache` for lazy loading of the `google/madlad400-3b-mt` tokenizer and model with `float16` precision (no download on import). Translation prepends a language token to the input text (e.g., `<2fr>Hello`) before tokenization and generation. The `@spaces.GPU` decorator allocates GPU on HF Spaces infrastructure.
36
 
37
- **`langmap/`** — Package with `langid_mapping.py`, a hand-maintained dictionary mapping ~400 language tokens to human-readable language names (sourced from pages 16–21 of the MADLAD-400 paper). Available languages at runtime are the intersection of this mapping and the model's vocabulary.
38
 
39
  **`tests/`** — Pytest suite split into fast (`test_langmap.py`) and slow (`test_app.py`). Slow tests require GPU and model download; they are auto-skipped without MPS/CUDA. Fast tests in `test_app.py` verify the module imports without triggering model download.
40
 
 
4
 
5
  ## Project Overview
6
 
7
+ A Hugging Face Spaces app that translates English text to 183 evaluated languages (Table 14, Section A.9) from Google's [MADLAD-400](https://arxiv.org/pdf/2309.04662) 3B Seq2Seq model. Built with Gradio and deployed on HF Spaces. Falls back to CPU with a warning when no GPU is available.
8
 
9
  ## Commands
10
 
 
34
 
35
  **`app.py`** — Single-file application. Uses `@lru_cache` for lazy loading of the `google/madlad400-3b-mt` tokenizer and model with `float16` precision (no download on import). Translation prepends a language token to the input text (e.g., `<2fr>Hello`) before tokenization and generation. The `@spaces.GPU` decorator allocates GPU on HF Spaces infrastructure.
36
 
37
+ **`langmap/`** — Package with `langid_mapping.py`, a hand-maintained dictionary mapping 183 evaluated language tokens to human-readable language names (sourced from Table 14, Section A.9 of the MADLAD-400 paper; evaluation sets: WMT, Flores-200, NTREX, Gatones). Available languages at runtime are the intersection of this mapping and the model's vocabulary.
38
 
39
  **`tests/`** — Pytest suite split into fast (`test_langmap.py`) and slow (`test_app.py`). Slow tests require GPU and model download; they are auto-skipped without MPS/CUDA. Fast tests in `test_app.py` verify the module imports without triggering model download.
40
 
README.md CHANGED
@@ -8,12 +8,12 @@ sdk_version: 6.8.0
8
  app_file: app.py
9
  pinned: false
10
  license: apache-2.0
11
- short_description: Translate English text to (almost) 400 languages.
12
  ---
13
 
14
  # MADLAD-400 Translation
15
 
16
- Translate English text to nearly 400 languages using Google's [MADLAD-400](https://arxiv.org/pdf/2309.04662) 3B model.
17
 
18
  ## Requirements
19
 
 
8
  app_file: app.py
9
  pinned: false
10
  license: apache-2.0
11
+ short_description: Translate English text to 183 evaluated languages.
12
  ---
13
 
14
  # MADLAD-400 Translation
15
 
16
+ Translate English text to 183 evaluated languages (Table 14, Section A.9) using Google's [MADLAD-400](https://arxiv.org/pdf/2309.04662) 3B model.
17
 
18
  ## Requirements
19