Tom9358 commited on
Commit
78c71c2
·
verified ·
1 Parent(s): 2864a0a

Update README.md in accordance with the latest model push.

Browse files
Files changed (1) hide show
  1. README.md +14 -11
README.md CHANGED
@@ -24,10 +24,11 @@ Moi!
24
 
25
  I used sentence pairs from https://tatoeba.org/ to finetune an NLLB model for Gronings. Consider this an early beta version!
26
 
27
- I am a linguist and speaker of Gronings so I carried out evaluation by expert's eyeball. The github repo contains some BLEU and ChrF score plots, but I haven't thoroughly investigated the performance by means of them and am hesitant to claim any particular general translation performance for this version.
28
- The model produces something that is recognizable as Gronings when the input language is Dutch. I found that interesting enough for a PoC, so I decided to publish.
29
-
30
- The model is not optimal in terms of hyperparameters, so I am planning to upload an even better version in the future.
 
31
 
32
  Update 10 September 2025: I've updated the code to the latest version of `transformers` so that it can immediately be used by anyone without any tokenizer black magic needed. Also about 500 more parallel nld-gos sentences were added to the training data.
33
  Only the additional Gronings language token needs to be added to the tokenizer at initialization, then everything should work.
@@ -36,6 +37,12 @@ Update 21 November 2025: I added another ~450 sentences and shortened training a
36
 
37
  Update 28 November 2025: Add another ~90 sentences. Upon further inspection, and inspired by https://www.youtube.com/watch?v=z64a7USuGX0 I decided to train for much longer, and this version seems to perform quite a bit better than the previous.
38
 
 
 
 
 
 
 
39
  <details>
40
  <summary>See here a minimal example code snippet to get the model up and running: (click)</summary>
41
 
@@ -56,12 +63,12 @@ def translate(text, src_lang: str = "nld_Latn", tgt_lang: str = "gos_Latn", **kw
56
  return_tensors='pt',
57
  padding='longest',
58
  truncation=True,
59
- max_length=120
60
  )
61
  result = model.generate(
62
  **inputs.to(model.device),
63
  forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
64
- max_new_tokens=int(16 + 1.5 * inputs.input_ids.shape[1]),
65
  **kwargs
66
  )
67
  return tokenizer.batch_decode(result, skip_special_tokens=True)
@@ -70,11 +77,7 @@ translate("Dit is een testzin om te kijken of de code werkt.")
70
  ```
71
  </details>
72
 
73
- See https://github.com/tom9358/nllb-tryout for everything (code, more documentation and references) except the model itself and training data.
74
-
75
- Check out the dedicated Huggingface space to try out the model!
76
-
77
- Alternatively, another (rather slow, but also free and accessible to everyone) way to try out the model:
78
  https://colab.research.google.com/drive/1b5dn3VT4fvOBKly1CIx4Qwo59GDM1H-M
79
 
80
  The code there is also a minimal example of how to use this model.
 
24
 
25
  I used sentence pairs from https://tatoeba.org/ to finetune an NLLB model for Gronings. Consider this an early beta version!
26
 
27
+ The github repo contains some BLEU and ChrF score plots, but I haven't thoroughly investigated the performance by means of them and am hesitant to claim any particular general translation performance for this version.
28
+ Fortunately, I am a linguist and speaker of Gronings so I could carry out evaluation by expert's eyeball.
29
+ The model generally produces acceptable Gronings when the input language is Dutch and the language is basic.
30
+ I consider this interesting enough for a public PoC, so I decided to publish.
31
+ It is certainly possible to create a better model by means of (synthetic) data addition and hyperparameter optimization. I have not employed backtranslation either yet.
32
 
33
  Update 10 September 2025: I've updated the code to the latest version of `transformers` so that it can immediately be used by anyone without any tokenizer black magic needed. Also about 500 more parallel nld-gos sentences were added to the training data.
34
  Only the additional Gronings language token needs to be added to the tokenizer at initialization, then everything should work.
 
37
 
38
  Update 28 November 2025: Add another ~90 sentences. Upon further inspection, and inspired by https://www.youtube.com/watch?v=z64a7USuGX0 I decided to train for much longer, and this version seems to perform quite a bit better than the previous.
39
 
40
+ Update 16 January 2026: Add a few hundred more sentences and allow longer sentences in training loop (at the cost of longer training time).
41
+
42
+ See https://github.com/tom9358/nllb-tryout for everything (code, more documentation and references) except the model itself and training data.
43
+
44
+ Check out the dedicated Huggingface space to try out the model! Find it here: https://huggingface.co/spaces/Tom9358/gos_gronings_translate
45
+
46
  <details>
47
  <summary>See here a minimal example code snippet to get the model up and running: (click)</summary>
48
 
 
63
  return_tensors='pt',
64
  padding='longest',
65
  truncation=True,
66
+ max_length=500
67
  )
68
  result = model.generate(
69
  **inputs.to(model.device),
70
  forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
71
+ max_new_tokens=int(20 + 1.6 * inputs.input_ids.shape[1]),
72
  **kwargs
73
  )
74
  return tokenizer.batch_decode(result, skip_special_tokens=True)
 
77
  ```
78
  </details>
79
 
80
+ In case for some reason the HF space stops working, here is another (albeit slower) way to try out the model:
 
 
 
 
81
  https://colab.research.google.com/drive/1b5dn3VT4fvOBKly1CIx4Qwo59GDM1H-M
82
 
83
  The code there is also a minimal example of how to use this model.