---
license: apache-2.0
language:
- ko
- en
base_model: facebook/wav2vec2-xls-r-300m
library_name: transformers
pipeline_tag: automatic-speech-recognition
tags:
- wav2vec2
- ctc
- korean
- english
- code-switching
---

# wav2vec2-xls-r-300m-korean-english

A **jamo + Latin CTC** acoustic model fine-tuned from `facebook/wav2vec2-xls-r-300m`
for **Korean / English code-switched** speech. It emits per-frame character
probabilities over a compact jamo + Latin vocabulary.

## Training

- **Base:** `facebook/wav2vec2-xls-r-300m` (300M multilingual wav2vec2), feature encoder frozen.
- **Head:** fresh CTC head over a ~105-token **jamo + Latin** vocab.
- **Data:** 17.4 h of clean TTS-synthesized KO/EN/mixed speech (~13.5k clips, 0.3–15 s),
  roughly 57% code-switched / 22% English / 21% Korean.
- **Recipe:** 15 epochs, lr 3e-4, batch 16, bf16, best checkpoint by CER.

## Important: jamo preprocessing

The vocab is **jamo (decomposed Hangul) + Latin**, not raw syllables. You MUST
decompose target text the same way, and recompose model output:

```python
import unicodedata
def to_jamo(t):   # text -> model target space (NFD on Hangul only)
    return "".join(unicodedata.normalize("NFD", c) if "가" <= c <= "힣" else c for c in t)
def from_jamo(t): # model output -> readable text
    return unicodedata.normalize("NFC", t)
```

## Usage

```python
import soundfile as sf, unicodedata
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

proc = Wav2Vec2Processor.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english")
model = Wav2Vec2ForCTC.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english").eval()

wav, sr = sf.read("clip.wav")  # resample to 16 kHz first if needed
x = proc(wav, sampling_rate=16000, return_tensors="pt").input_values
ids = model(x).logits.argmax(-1)[0]
print(unicodedata.normalize("NFC", proc.decode(ids)))
```