Fix Tokenizer
#7
by johnchen95 - opened
The tokenizer has been fixed for the Korean -> English model.
- https://huggingface.co/Helsinki-NLP/opus-mt-tc-big-ko-en/discussions/5
- https://github.com/Helsinki-NLP/OPUS-MT-train/issues/81
However, the same change was never applied to the English -> Korean model.
This Pull Request applies that same fix here.
English:
Personal transportation devices (skateboards, hover boards, scooters, etc.)
Korean:
개인 수송 장치 (스케이트 보드, 호버 보드, 스쿠터 등)
johnchen95 changed pull request title from Upload 2 files to Fix Tokenizer