Hugging Face's logo

LocalDoc
/
nllb-3.3b-en-az

Translation
Transformers
Safetensors
English
Azerbaijani
m2m_100
text2text-generation
azerbaijani
nllb
english-azerbaijani
Model card Files
xet
 Community
vrashad commited on
Commit
24effe3
·
verified ·
1 Parent(s): 977c87f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -16
README.md CHANGED
@@ -24,22 +24,22 @@ On a 1,012-sentence FLORES-based benchmark it **outperforms Google Translate and
24
  ## Benchmark Results
25
 
26
  Evaluated on [`LocalDoc/en_az_translate_benchmark`](https://huggingface.co/datasets/LocalDoc/en_az_translate_benchmark) (1,012 sentences, EN→AZ). All metrics are reference-based: chrF++ (`word_order=2`), BLEU (sacreBLEU), and COMET-DA (`Unbabel/wmt22-comet-da`). Higher is better.
27
-
28
- | Model | chrF++ | BLEU | COMET-DA |
29
- |---|---|---|---|
30
- | GPT-5.4-mini | 70.08 | 45.61 | 92.86 |
31
- | Gemini-2.5-flash | 69.61 | 45.71 | 92.70 |
32
- | **This model (NLLB-3.3B EN→AZ)** | **69.30** | **44.42** | **92.70** |
33
- | DeepSeek-V4-Pro | 68.67 | 43.88 | 92.78 |
34
- | DeepSeek-V4-Flash | 67.96 | 42.82 | 92.58 |
35
- | GPT-4.1 | 67.76 | 43.03 | 92.71 |
36
- | Google Translate | 66.90 | 41.64 | 92.37 |
37
- | Gemma-4-31B-it | 66.22 | 40.46 | 92.40 |
38
- | GPT-5.4-nano | 62.10 | 33.87 | 91.41 |
39
- | Qwen3.6-35B-A3B | 60.39 | 33.57 | 91.23 |
40
- | NLLB-200-3.3B (base, zero-shot) | 59.03 | 31.76 | 89.86 |
41
-
42
- Fine-tuning improved the base NLLB-3.3B by **+10.3 chrF++** (59.03 → 69.30) and **+2.84 COMET-DA** (89.86 → 92.70). The result surpasses Google Translate, DeepSeek-V4-Pro, GPT-4.1, Gemma-3-31B, and Qwen-35B, and comes within 0.8 chrF++ of GPT-5.4-mini and Gemini-2.5-flash.
43
 
44
  ## Example Translations
45
 
 
24
  ## Benchmark Results
25
 
26
  Evaluated on [`LocalDoc/en_az_translate_benchmark`](https://huggingface.co/datasets/LocalDoc/en_az_translate_benchmark) (1,012 sentences, EN→AZ). All metrics are reference-based: chrF++ (`word_order=2`), BLEU (sacreBLEU), and COMET-DA (`Unbabel/wmt22-comet-da`). Higher is better.
27
+ |
28
+ | Model | chrF++ | BLEU | COMET-DA |
29
+ | -------------------------------- | --------: | --------: | --------: |
30
+ | GPT-5.4-mini | 70.08 | 45.61 | 92.86 |
31
+ | **This model (NLLB-3.3B EN→AZ)** | **70.00** | **45.86** | **92.75** |
32
+ | Gemini-2.5-flash | 69.61 | 45.71 | 92.70 |
33
+ | DeepSeek-V4-Pro | 68.67 | 43.88 | 92.78 |
34
+ | DeepSeek-V4-Flash | 67.96 | 42.82 | 92.58 |
35
+ | GPT-4.1 | 67.76 | 43.03 | 92.71 |
36
+ | Google Translate | 66.90 | 41.64 | 92.37 |
37
+ | Gemma-4-31B-it | 66.22 | 40.46 | 92.40 |
38
+ | GPT-5.4-nano | 62.10 | 33.87 | 91.41 |
39
+ | Qwen3.6-35B-A3B | 60.39 | 33.57 | 91.23 |
40
+ | NLLB-200-3.3B (base, zero-shot) | 59.03 | 31.76 | 89.86 |
41
+
42
+ Fine-tuning improved the base NLLB-3.3B by **+10.97 chrF++** (59.03 → 69.30) and **+2.89 COMET-DA** (89.86 → 92.75). The result surpasses Google Translate, DeepSeek-V4-Pro, GPT-4.1, Gemma-3-31B, and Qwen-35B, and comes within 0.8 chrF++ of GPT-5.4-mini and Gemini-2.5-flash.
43
 
44
  ## Example Translations
45