wav2vec2-xls-r-300m-korean-english

A jamo + Latin CTC acoustic model fine-tuned from facebook/wav2vec2-xls-r-300m for Korean / English code-switched speech. It emits per-frame character probabilities over a compact jamo + Latin vocabulary.

Training

• Base: facebook/wav2vec2-xls-r-300m (300M multilingual wav2vec2), feature encoder frozen.
• Head: fresh CTC head over a ~105-token jamo + Latin vocab.
• Data: 17.4 h of clean TTS-synthesized KO/EN/mixed speech (~13.5k clips, 0.3–15 s), roughly 57% code-switched / 22% English / 21% Korean.
• Recipe: 15 epochs, lr 3e-4, batch 16, bf16, best checkpoint by CER.

Important: jamo preprocessing

The vocab is jamo (decomposed Hangul) + Latin, not raw syllables. You MUST decompose target text the same way, and recompose model output:

import unicodedata
def to_jamo(t):   # text -> model target space (NFD on Hangul only)
    return "".join(unicodedata.normalize("NFD", c) if "가" <= c <= "힣" else c for c in t)
def from_jamo(t): # model output -> readable text
    return unicodedata.normalize("NFC", t)

Usage

import soundfile as sf, unicodedata
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

proc = Wav2Vec2Processor.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english")
model = Wav2Vec2ForCTC.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english").eval()

wav, sr = sf.read("clip.wav")  # resample to 16 kHz first if needed
x = proc(wav, sampling_rate=16000, return_tensors="pt").input_values
ids = model(x).logits.argmax(-1)[0]
print(unicodedata.normalize("NFC", proc.decode(ids)))