madlad-400-translate

Running

Daryl Lim Claude Opus 4.6 commited on Mar 2

Commit

7d583b2

1 Parent(s): 613b55a

fix: add space between language token and input text

MADLAD-400 expects `<2fr> Hello` not `<2fr>Hello`. Without the
space, the tokenizer fails to recognize the language token,
causing garbage repetition in the output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (1) hide show

app.py +1 -1

app.py CHANGED Viewed

@@ -57,7 +57,7 @@ def translate(text: str, target_language_name: str) -> str:
     if target_code is None:
         raise ValueError(f"Unsupported language: {target_language_name}")
-    input_ids = tokenizer(target_code + text, return_tensors="pt").input_ids.to(device)
     outputs = model.generate(input_ids=input_ids, max_new_tokens=512)
     result = tokenizer.decode(outputs[0], skip_special_tokens=True)
     if not isinstance(result, str):

     if target_code is None:
         raise ValueError(f"Unsupported language: {target_language_name}")
+    input_ids = tokenizer(target_code + " " + text, return_tensors="pt").input_ids.to(device)
     outputs = model.generate(input_ids=input_ids, max_new_tokens=512)
     result = tokenizer.decode(outputs[0], skip_special_tokens=True)
     if not isinstance(result, str):