---
license: cc-by-nc-4.0
library_name: transformers
tags:
- speech
- audio
- multimodal
- mixture-of-experts
- text-to-speech
- automatic-speech-recognition
language:
- en
- ko
pipeline_tag: any-to-any
---
# A.X K2 Raon-Speech
English | 한국어
A.X K2 Raon-Speech is a bilingual English/Korean speech language model with
approximately 21.2B total parameters and 3.5B active parameters. Built on
SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts text backbone, it
integrates an AuT speech encoder and a Mimi-style neural audio codec
independently trained by KRAFTON AI, unifying speech understanding and
generation in a single multimodal model. It is trained with the recipe of
[Raon-Speech-9B](https://huggingface.co/KRAFTON/Raon-Speech-9B), KRAFTON's
open-source speech language model.
## Key Features
- **End-to-End Speech Language Model**: 21.2B-parameter (approximately 3.5B
active) multimodal model built on SK Telecom's A.X K2 Light 20B-A3B
mixture-of-experts backbone.
- **Independently Trained Speech Modules**: AuT speech encoder
(approximately 317M parameters) and Mimi-style neural audio codec
(approximately 96M parameters), trained on curated data with an in-house
pipeline.
- **Bilingual Support**: First in Korean and third in English aggregate score
among publicly available speech language models at or below the 30B scale.
- **Multi-Task Capabilities**: STT, TTS, SpeechQA, SpokenQA, and turn-based
multimodal chat in a single unified model.
- **Paralinguistic Awareness**: Uses emotion, intonation, and other
information carried in the user's voice to generate natural spoken
responses.
- **Speaker Voice Conditioning**: TTS with optional speaker reference audio
via ECAPA embeddings.
- **TTS Continuation**: Generates speech that naturally continues from a
reference audio with prefill-based continuation for seamless prosody.
## Architecture
| Component | Configuration |
|---|---|
| Text backbone | A.X K2 Light 20B-A3B MoE, 48 layers, 2,048 hidden size |
| Experts | 128 routed experts, top-8 routing |
| Speech encoder | Independently trained AuT encoder based on the Qwen3-ASR architecture, 24 layers, approximately 317M parameters |
| Speech codec | Independently trained Mimi-style codec, approximately 96M parameters |
| Codec configuration | 32 quantizers, 2,048-entry codebooks, 24 kHz, 12.5 frames per second |
| SpeechLM code interface | 8 input code groups and 16 generated output code groups |
| Talker | Qwen3-based talker, 4 layers, 2,048 hidden size |
| Text-backbone context window | Up to 131,072 tokens |
All components for the core speech-understanding and direct-TTS paths are
included in the checkpoint. Speaker-conditioned and continuation TTS
additionally use a frozen SpeechBrain ECAPA speaker encoder, downloaded
automatically on first use.
## Evaluation
Evaluation covered six task areas — speech recognition, speech synthesis,
speech understanding, spoken question answering, text question answering, and
tool calling — using 46 benchmarks (24 Korean, 22 English), including
LibriSpeech, KsponSpeech, VoiceBench, KVoiceBench, MMAU, KMMAU, API-Bank, and
FunctionChat-Bench. Each result is normalized to the 0–1 range and aggregated
per language; see the
[KRAFTON AI tech blog](https://krafton.ai/blog/posts/2026-06-30-ax-raon/ax-raon-en.html)
for the full methodology and results.
Among publicly available speech language models at or below the 30B scale,
A.X K2 Raon-Speech ranked first on the Korean aggregate score (0.72) and
third on the English aggregate score (0.75).
Per-category performance, normalized from zero to the highest score in each
category. Comparison models are the top English performers below that support
both English and Korean.
### Aggregate scores
| Korean Rank | Model | Korean | English | English Rank |
|---|---|---|---|---|
| 1 | **A.X K2 Raon-Speech** | **0.72** | **0.75** | **3** |
| 2 | Raon-Speech | 0.70 | 0.79 | 1 |
| 3 | Qwen3-Omni | 0.65 | 0.78 | 2 |
| 4 | HyperCLOVA X 8B Omni | 0.60 | 0.68 | 8 |
| 5 | MOSS-Audio | 0.58 | 0.62 | 13 |
| 6 | Qwen2.5-Omni | 0.55 | 0.73 | 5 |
| 7 | Fun-Audio-Chat | 0.53 | 0.69 | 7 |
| 8 | Step-Audio 2 mini | 0.32 | 0.67 | 10 |
| 9 | Interactive-Omni | 0.27 | 0.71 | 6 |
| 10 | Ming-Lite-Omni v1.5 | 0.21 | 0.68 | 8 |
| 10 | AF-Next | 0.21 | 0.65 | 11 |
| 12 | COVO-Audio | 0.18 | 0.41 | 15 |
| — | MiniCPM-o 4.5 | — | 0.74 | 4 |
| — | Kimi-Audio | — | 0.65 | 11 |
| — | Audio Flamingo 3 | — | 0.60 | 14 |
### Speech encoder
The AuT speech encoder matches Qwen3-ASR AuT on SpeechLM subtasks while
trained on approximately 1M hours of audio versus the 40M hours reported for
Qwen3-ASR-1.7B AuT (values relative to Qwen AuT = 100%).
| Task / Metric | English | Korean | Language Average |
|---|---|---|---|
| Speech Recognition | 96.7% | 104.6% | 100.7% |
| SpokenQA (Accuracy) | 90.3% | 104.8% | 97.6% |
| SpeechQA (Accuracy) | 98.8% | 100.2% | 99.5% |
| **Overall Language Average** | **95.3%** | **103.2%** | **99.2%** |
### Speech codec
The codec approaches Mimi's performance (values relative to Mimi = 100%)
while trained on approximately 500K hours of audio; Moshi reports a 7M-hour
unsupervised corpus for audio pretraining.
| Metric | English | Korean | Language Average |
|---|---|---|---|
| PESQ | 83.7% | 83.3% | 83.5% |
| ESTOI | 96.8% | 94.7% | 95.8% |
| Speech Recognition | 100.7% | 112.6% | 106.7% |
| Speaker Similarity | 96.9% | 99.6% | 98.3% |
| UTMOS | 94.4% | 95.7% | 95.1% |
| **Overall Language Average** | **94.5%** | **97.2%** | **95.9%** |
### Post-training ablations
Against the immediately preceding checkpoint, tool-use fine-tuning improved
the tool-call score from 0.543 to 0.740, paralinguistic self-distillation
improved the emotion/speaker score from 0.566 to 0.613, and TTS direct
preference optimization reduced the average TTS error rate from 3.6% to 3.1%.
## Requirements
```bash
pip install -r requirements.txt
# Optional: lower-latency attention on supported CUDA systems
pip install flash-attn --no-build-isolation
```
Requires `transformers>=4.57.1`. `speechbrain` is required for the
speaker-conditioned and continuation TTS APIs.
## Load the model
```python
from transformers import AutoConfig
from transformers.dynamic_module_utils import get_class_from_dynamic_module
MODEL_ID = "KRAFTON/A.X-K2-Raon-Speech-21B-A3B"
config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
RaonPipeline = get_class_from_dynamic_module(
"modeling_raon.RaonPipeline",
MODEL_ID,
revision=getattr(config, "_commit_hash", None),
)
pipe = RaonPipeline(
MODEL_ID,
device="cuda",
dtype="bfloat16",
)
```
The BF16 checkpoint holds approximately 42.4 GB of weights; an 80 GB GPU or a
suitable multi-GPU configuration is recommended.
## Usage
### Speech-to-text
```python
text = pipe.stt("audio.wav")
```
### Text-to-speech
```python
audio, sample_rate = pipe.tts("Hello, how are you?")
pipe.save_audio((audio, sample_rate), "output.wav")
```
### Speaker-conditioned TTS
```python
audio, sample_rate = pipe.tts(
"안녕하세요. 만나서 반갑습니다.",
speaker_audio="speaker_reference.wav",
)
pipe.save_audio((audio, sample_rate), "conditioned.wav")
```
### TTS continuation from reference audio
```python
audio, sample_rate = pipe.tts_continuation(
"And this is how the story continues.",
ref_audio="reference.wav",
)
pipe.save_audio((audio, sample_rate), "continuation.wav")
```
### SpeechQA (audio context, text question)
```python
messages = [
{
"role": "user",
"content": [
{"type": "audio", "audio": "context.wav"},
{"type": "text", "text": "Please answer the question in this audio."},
],
}
]
response = pipe.chat(messages)
```
### SpokenQA (audio context, audio question)
```python
messages = [
{
"role": "user",
"content": [
{"type": "audio", "audio": "spoken_question.wav"},
],
}
]
response = pipe.chat(messages)
```
## Deployment with vLLM-Omni
Use KRAFTON's official
[vLLM-Omni repository](https://github.com/krafton-ai/vllm-omni) and the
compatible `feat/axk2_raon_speech` branch. The validated revision is
[`7bcaf21`](https://github.com/krafton-ai/vllm-omni/commit/7bcaf21df881bda4280cbbf9752fc356e753d3bb);
you can optionally pin it to match the vLLM-Omni implementation revision used
for validation.
The online server supports these request patterns:
| Capability | Request | Response |
| --- | --- | --- |
| Direct or speaker-conditioned TTS / voice cloning | Text, optionally with reference audio | WAV from `/v1/audio/speech` |
| STT | Audio with a written transcription instruction | Text from `/v1/chat/completions` |
| TextQA | Written question | Text answer from `/v1/chat/completions` |
| SpeechQA / multimodal chat | Audio context with a written question or instruction | Text answer from `/v1/chat/completions` |
| SpokenQA / SpeechChat | Spoken audio question without a written prompt | Text answer from `/v1/chat/completions` |
### Install and serve
```bash
git clone --branch feat/axk2_raon_speech --single-branch \
https://github.com/krafton-ai/vllm-omni.git
cd vllm-omni
# Optional: pin the revision validated with this checkpoint.
git checkout 7bcaf21df881bda4280cbbf9752fc356e753d3bb
python -m pip install "vllm==0.20.0"
python -m pip install -e .
```
The bundled stage profile targets one GPU with approximately 96 GB of VRAM.
It uses tensor parallelism 1 (`TP=1`) and sets both the maximum model length
and maximum batched tokens to 4096. With vLLM 0.20,
`VLLM_MLA_DISABLE=1` is required because this checkpoint has a 192-wide
KV-cache head.
From the cloned repository root, start the server with:
```bash
export VLLM_MLA_DISABLE=1
vllm-omni serve KRAFTON/A.X-K2-Raon-Speech-21B-A3B \
--stage-configs-path examples/offline_inference/raon/raon_axk1_local.yaml \
--host 0.0.0.0 \
--port 8091 \
--trust-remote-code \
--omni
```
### Health and model listing
```bash
curl -fS http://localhost:8091/health
curl -fS http://localhost:8091/v1/models
```
### TTS
```bash
curl -fS -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
--data-binary @- \
--output axk2_tts.wav <<'EOF'
{
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
"input": "Hello, this is A.X-K2 Raon Speech.",
"response_format": "wav"
}
EOF
```
### Speaker-conditioned TTS / voice cloning
Encode the reference speaker audio as a portable single-line base64 data URL:
```bash
REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"
curl -fS -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
--data-binary @- \
--output axk2_conditioned.wav <
