README.md

---
license: cc-by-nc-4.0
library_name: transformers
tags:
- speech
- audio
- multimodal
- mixture-of-experts
- text-to-speech
- automatic-speech-recognition
language:
- en
- ko
---

<div align="center">
  <img class="block dark:hidden" src="assets/AX_K2_Raon_Logo_Black.png" alt="A.X-K2-Raon Logo" width="80%">
  <img class="hidden dark:block" src="assets/AX_K2_Raon_Logo_White.png" alt="A.X-K2-Raon Logo" width="80%">
</div>

# A.X K2 Raon-Speech

<p align="center">
  <a href="https://new.krafton.ai/en/"><img src="https://img.shields.io/badge/Homepage-KRAFTON%20AI-blue?style=flat&logo=google-chrome&logoColor=white" alt="Homepage"></a>
  <a href="https://krafton.ai/blog/posts/2026-06-30-ax-raon/ax-raon-en.html"><img src="https://img.shields.io/badge/Tech%20Blog-A.X%20K2%20Raon-orange?style=flat&logo=rss&logoColor=white" alt="Tech Blog"></a>
  <a href="https://github.com/krafton-ai/Raon-Speech"><img src="https://img.shields.io/badge/GitHub-Raon-white?style=flat&logo=github&logoColor=black" alt="GitHub"></a>
  <a href="https://huggingface.co/KRAFTON"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-KRAFTON-yellow?style=flat" alt="Hugging Face"></a>
  <a href="https://creativecommons.org/licenses/by-nc/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey?style=flat" alt="License"></a>
</p>

<p align="center">
  <b>English</b> | <a href="https://huggingface.co/KRAFTON/A.X-K2-Raon-Speech-21B-A3B/blob/main/README_ko.md">한국어</a>
</p>

A.X K2 Raon-Speech is a bilingual English/Korean speech language model with
approximately 21.2B total parameters and 3.5B active parameters. Built on
SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts text backbone, it
integrates an AuT speech encoder and a Mimi-style neural audio codec
independently trained by KRAFTON AI, unifying speech understanding and
generation in a single multimodal model. It is trained with the recipe of
[Raon-Speech-9B](https://huggingface.co/KRAFTON/Raon-Speech-9B), KRAFTON's
open-source speech language model.

## Key Features

- **End-to-End Speech Language Model**: 21.2B-parameter (approximately 3.5B
  active) multimodal model built on SK Telecom's A.X K2 Light 20B-A3B
  mixture-of-experts backbone.
- **Independently Trained Speech Modules**: AuT speech encoder
  (approximately 317M parameters) and Mimi-style neural audio codec
  (approximately 96M parameters), trained on curated data with an in-house
  pipeline.
- **Bilingual Support**: First in Korean and third in English aggregate score
  among publicly available speech language models at or below the 30B scale.
- **Multi-Task Capabilities**: STT, TTS, SpeechQA, SpokenQA, and turn-based
  multimodal chat in a single unified model.
- **Paralinguistic Awareness**: Uses emotion, intonation, and other
  information carried in the user's voice to generate natural spoken
  responses.
- **Speaker Voice Conditioning**: TTS with optional speaker reference audio
  via ECAPA embeddings.
- **TTS Continuation**: Generates speech that naturally continues from a
  reference audio with prefill-based continuation for seamless prosody.

## Architecture

| Component | Configuration |
|---|---|
| Text backbone | A.X K2 Light 20B-A3B MoE, 48 layers, 2,048 hidden size |
| Experts | 128 routed experts, top-8 routing |
| Speech encoder | Independently trained AuT encoder based on the Qwen3-ASR architecture, 24 layers, approximately 317M parameters |
| Speech codec | Independently trained Mimi-style codec, approximately 96M parameters |
| Codec configuration | 32 quantizers, 2,048-entry codebooks, 24 kHz, 12.5 frames per second |
| SpeechLM code interface | 8 input code groups and 16 generated output code groups |
| Talker | Qwen3-based talker, 4 layers, 2,048 hidden size |
| Text-backbone context window | Up to 131,072 tokens |

All components for the core speech-understanding and direct-TTS paths are
included in the checkpoint. Speaker-conditioned and continuation TTS
additionally use a frozen SpeechBrain ECAPA speaker encoder, downloaded
automatically on first use.

## Evaluation

Evaluation covered six task areas — speech recognition, speech synthesis,
speech understanding, spoken question answering, text question answering, and
tool calling — using 46 benchmarks (24 Korean, 22 English), including
LibriSpeech, KsponSpeech, VoiceBench, KVoiceBench, MMAU, KMMAU, API-Bank, and
FunctionChat-Bench. Each result is normalized to the 0–1 range and aggregated
per language; see the
[KRAFTON AI tech blog](https://krafton.ai/blog/posts/2026-06-30-ax-raon/ax-raon-en.html)
for the full methodology and results.

Among publicly available speech language models at or below the 30B scale,
A.X K2 Raon-Speech ranked first on the Korean aggregate score (0.72) and
third on the English aggregate score (0.75).

<div align="center">
  <img src="assets/ax-k2-raon-speech-fig1-radar.png" alt="A.X K2 Raon-Speech per-category benchmark radar" width="800">
</div>

Per-category performance, normalized from zero to the highest score in each
category. Comparison models are the top English performers below that support
both English and Korean.

### Aggregate scores

| Korean Rank | Model | Korean | English | English Rank |
|---|---|---|---|---|
| 1 | **A.X K2 Raon-Speech** | **0.72** | **0.75** | **3** |
| 2 | Raon-Speech | 0.70 | 0.79 | 1 |
| 3 | Qwen3-Omni | 0.65 | 0.78 | 2 |
| 4 | HyperCLOVA X 8B Omni | 0.60 | 0.68 | 8 |
| 5 | MOSS-Audio | 0.58 | 0.62 | 13 |
| 6 | Qwen2.5-Omni | 0.55 | 0.73 | 5 |
| 7 | Fun-Audio-Chat | 0.53 | 0.69 | 7 |
| 8 | Step-Audio 2 mini | 0.32 | 0.67 | 10 |
| 9 | Interactive-Omni | 0.27 | 0.71 | 6 |
| 10 | Ming-Lite-Omni v1.5 | 0.21 | 0.68 | 8 |
| 10 | AF-Next | 0.21 | 0.65 | 11 |
| 12 | COVO-Audio | 0.18 | 0.41 | 15 |
| — | MiniCPM-o 4.5 | — | 0.74 | 4 |
| — | Kimi-Audio | — | 0.65 | 11 |
| — | Audio Flamingo 3 | — | 0.60 | 14 |

### Speech encoder

The AuT speech encoder matches Qwen3-ASR AuT on SpeechLM subtasks while
trained on approximately 1M hours of audio versus the 40M hours reported for
Qwen3-ASR-1.7B AuT (values relative to Qwen AuT = 100%).

| Task / Metric | English | Korean | Language Average |
|---|---|---|---|
| Speech Recognition | 96.7% | 104.6% | 100.7% |
| SpokenQA (Accuracy) | 90.3% | 104.8% | 97.6% |
| SpeechQA (Accuracy) | 98.8% | 100.2% | 99.5% |
| **Overall Language Average** | **95.3%** | **103.2%** | **99.2%** |

### Speech codec

The codec approaches Mimi's performance (values relative to Mimi = 100%)
while trained on approximately 500K hours of audio; Moshi reports a 7M-hour
unsupervised corpus for audio pretraining.

| Metric | English | Korean | Language Average |
|---|---|---|---|
| PESQ | 83.7% | 83.3% | 83.5% |
| ESTOI | 96.8% | 94.7% | 95.8% |
| Speech Recognition | 100.7% | 112.6% | 106.7% |
| Speaker Similarity | 96.9% | 99.6% | 98.3% |
| UTMOS | 94.4% | 95.7% | 95.1% |
| **Overall Language Average** | **94.5%** | **97.2%** | **95.9%** |

### Post-training ablations

<div align="center">
  <img src="assets/ax-k2-ablation-combined.png" alt="Post-training ablation results" width="800">
</div>

Against the immediately preceding checkpoint, tool-use fine-tuning improved
the tool-call score from 0.543 to 0.740, paralinguistic self-distillation
improved the emotion/speaker score from 0.566 to 0.613, and TTS direct
preference optimization reduced the average TTS error rate from 3.6% to 3.1%.

## Requirements

```bash
pip install -r requirements.txt

# Optional: lower-latency attention on supported CUDA systems
pip install flash-attn --no-build-isolation
```

Requires `transformers>=4.57.1`. `speechbrain` is required for the
speaker-conditioned and continuation TTS APIs.

## Load the model

```python
from transformers import AutoConfig
from transformers.dynamic_module_utils import get_class_from_dynamic_module

MODEL_ID = "KRAFTON/A.X-K2-Raon-Speech-21B-A3B"

config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
RaonPipeline = get_class_from_dynamic_module(
    "modeling_raon.RaonPipeline",
    MODEL_ID,
    revision=getattr(config, "_commit_hash", None),
)

pipe = RaonPipeline(
    MODEL_ID,
    device="cuda",
    dtype="bfloat16",
)
```

The BF16 checkpoint holds approximately 42.4 GB of weights; an 80 GB GPU or a
suitable multi-GPU configuration is recommended.

## Usage

### Speech-to-text

```python
text = pipe.stt("audio.wav")
```

### Text-to-speech

```python
audio, sample_rate = pipe.tts("Hello, how are you?")
pipe.save_audio((audio, sample_rate), "output.wav")
```

### Speaker-conditioned TTS

```python
audio, sample_rate = pipe.tts(
    "안녕하세요. 만나서 반갑습니다.",
    speaker_audio="speaker_reference.wav",
)
pipe.save_audio((audio, sample_rate), "conditioned.wav")
```

### TTS continuation from reference audio

```python
audio, sample_rate = pipe.tts_continuation(
    "And this is how the story continues.",
    ref_audio="reference.wav",
)
pipe.save_audio((audio, sample_rate), "continuation.wav")
```

### SpeechQA (audio context, text question)

```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "context.wav"},
            {"type": "text", "text": "Please answer the question in this audio."},
        ],
    }
]

response = pipe.chat(messages)
```

### SpokenQA (audio context, audio question)

```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "spoken_question.wav"},
        ],
    }
]

response = pipe.chat(messages)
```

## Training

The released checkpoint is the final model following:

1. Speech-module alignment with the A.X K2 Light backbone frozen
2. Joint multimodal supervised fine-tuning with text-task mixing
3. Text and cross-modal distillation from A.X K2 Light
4. Speech-conditioned tool-use fine-tuning
5. ParaBridge-style on-policy self-distillation for paralinguistic behavior
6. TTS direct preference optimization, targeting elongated or repetitive
   syllables, incomplete termination, and recognition errors in generated
   speech

Repository packaging and the inference APIs were validated end-to-end,
including a 12-call smoke matrix covering English and Korean STT, direct TTS,
speaker-conditioned TTS, continuation TTS, SpeechQA, and SpokenQA:

```bash
python validate_repo.py
python validate_model.py
```

## Intended use

This checkpoint is intended for research and non-commercial prototyping in
bilingual speech recognition and generation, spoken question answering, audio
understanding, and speech-enabled assistants. Users are responsible for
obtaining consent before voice conditioning or generation involving
identifiable speakers.

## Acknowledgments

This work was supported by the Ministry of Science and ICT (MSIT), Republic of
Korea, through the National IT Industry Promotion Agency (NIPA) (Grant No.
PJT-26-010018). This research was also conducted as part of the Sovereign AI
Foundation Model Project (Data Track), organized by the Ministry of Science and
ICT (MSIT) and supported by the National Information Society Agency (NIA),
Republic of Korea (Grant No. 2026-AIData-WII01).

## License

This model is released under the
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. The
bundled A.X K2 Light implementation code retains its Apache License 2.0
notices, which apply to that code only.