Translation
Transformers
Safetensors
m2m_100
text2text-generation
language
linguistics
low-resource
tatoeba
nllb
machine-translation
gronings
Instructions to use Tom9358/nllb-tatoeba-gos-nld-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Tom9358/nllb-tatoeba-gos-nld-v1 with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # 'pip install "transformers<5.0.0' from transformers import pipeline pipe = pipeline("translation", model="Tom9358/nllb-tatoeba-gos-nld-v1")# Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("Tom9358/nllb-tatoeba-gos-nld-v1") model = AutoModelForSeq2SeqLM.from_pretrained("Tom9358/nllb-tatoeba-gos-nld-v1", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md in accordance with the latest model push.
Browse files
README.md
CHANGED
|
@@ -24,10 +24,11 @@ Moi!
|
|
| 24 |
|
| 25 |
I used sentence pairs from https://tatoeba.org/ to finetune an NLLB model for Gronings. Consider this an early beta version!
|
| 26 |
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
|
|
|
| 31 |
|
| 32 |
Update 10 September 2025: I've updated the code to the latest version of `transformers` so that it can immediately be used by anyone without any tokenizer black magic needed. Also about 500 more parallel nld-gos sentences were added to the training data.
|
| 33 |
Only the additional Gronings language token needs to be added to the tokenizer at initialization, then everything should work.
|
|
@@ -36,6 +37,12 @@ Update 21 November 2025: I added another ~450 sentences and shortened training a
|
|
| 36 |
|
| 37 |
Update 28 November 2025: Add another ~90 sentences. Upon further inspection, and inspired by https://www.youtube.com/watch?v=z64a7USuGX0 I decided to train for much longer, and this version seems to perform quite a bit better than the previous.
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
<details>
|
| 40 |
<summary>See here a minimal example code snippet to get the model up and running: (click)</summary>
|
| 41 |
|
|
@@ -56,12 +63,12 @@ def translate(text, src_lang: str = "nld_Latn", tgt_lang: str = "gos_Latn", **kw
|
|
| 56 |
return_tensors='pt',
|
| 57 |
padding='longest',
|
| 58 |
truncation=True,
|
| 59 |
-
max_length=
|
| 60 |
)
|
| 61 |
result = model.generate(
|
| 62 |
**inputs.to(model.device),
|
| 63 |
forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
|
| 64 |
-
max_new_tokens=int(
|
| 65 |
**kwargs
|
| 66 |
)
|
| 67 |
return tokenizer.batch_decode(result, skip_special_tokens=True)
|
|
@@ -70,11 +77,7 @@ translate("Dit is een testzin om te kijken of de code werkt.")
|
|
| 70 |
```
|
| 71 |
</details>
|
| 72 |
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
Check out the dedicated Huggingface space to try out the model!
|
| 76 |
-
|
| 77 |
-
Alternatively, another (rather slow, but also free and accessible to everyone) way to try out the model:
|
| 78 |
https://colab.research.google.com/drive/1b5dn3VT4fvOBKly1CIx4Qwo59GDM1H-M
|
| 79 |
|
| 80 |
The code there is also a minimal example of how to use this model.
|
|
|
|
| 24 |
|
| 25 |
I used sentence pairs from https://tatoeba.org/ to finetune an NLLB model for Gronings. Consider this an early beta version!
|
| 26 |
|
| 27 |
+
The github repo contains some BLEU and ChrF score plots, but I haven't thoroughly investigated the performance by means of them and am hesitant to claim any particular general translation performance for this version.
|
| 28 |
+
Fortunately, I am a linguist and speaker of Gronings so I could carry out evaluation by expert's eyeball.
|
| 29 |
+
The model generally produces acceptable Gronings when the input language is Dutch and the language is basic.
|
| 30 |
+
I consider this interesting enough for a public PoC, so I decided to publish.
|
| 31 |
+
It is certainly possible to create a better model by means of (synthetic) data addition and hyperparameter optimization. I have not employed backtranslation either yet.
|
| 32 |
|
| 33 |
Update 10 September 2025: I've updated the code to the latest version of `transformers` so that it can immediately be used by anyone without any tokenizer black magic needed. Also about 500 more parallel nld-gos sentences were added to the training data.
|
| 34 |
Only the additional Gronings language token needs to be added to the tokenizer at initialization, then everything should work.
|
|
|
|
| 37 |
|
| 38 |
Update 28 November 2025: Add another ~90 sentences. Upon further inspection, and inspired by https://www.youtube.com/watch?v=z64a7USuGX0 I decided to train for much longer, and this version seems to perform quite a bit better than the previous.
|
| 39 |
|
| 40 |
+
Update 16 January 2026: Add a few hundred more sentences and allow longer sentences in training loop (at the cost of longer training time).
|
| 41 |
+
|
| 42 |
+
See https://github.com/tom9358/nllb-tryout for everything (code, more documentation and references) except the model itself and training data.
|
| 43 |
+
|
| 44 |
+
Check out the dedicated Huggingface space to try out the model! Find it here: https://huggingface.co/spaces/Tom9358/gos_gronings_translate
|
| 45 |
+
|
| 46 |
<details>
|
| 47 |
<summary>See here a minimal example code snippet to get the model up and running: (click)</summary>
|
| 48 |
|
|
|
|
| 63 |
return_tensors='pt',
|
| 64 |
padding='longest',
|
| 65 |
truncation=True,
|
| 66 |
+
max_length=500
|
| 67 |
)
|
| 68 |
result = model.generate(
|
| 69 |
**inputs.to(model.device),
|
| 70 |
forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
|
| 71 |
+
max_new_tokens=int(20 + 1.6 * inputs.input_ids.shape[1]),
|
| 72 |
**kwargs
|
| 73 |
)
|
| 74 |
return tokenizer.batch_decode(result, skip_special_tokens=True)
|
|
|
|
| 77 |
```
|
| 78 |
</details>
|
| 79 |
|
| 80 |
+
In case for some reason the HF space stops working, here is another (albeit slower) way to try out the model:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
https://colab.research.google.com/drive/1b5dn3VT4fvOBKly1CIx4Qwo59GDM1H-M
|
| 82 |
|
| 83 |
The code there is also a minimal example of how to use this model.
|