README.md

metadata

license: cc-by-nc-4.0
library_name: transformers
tags:
  - speech
  - audio
  - multimodal
  - mixture-of-experts
  - text-to-speech
  - automatic-speech-recognition
language:
  - en
  - ko

A.X K2 Raon-Speech

English | 한국어

A.X K2 Raon-Speech is a bilingual English/Korean speech language model with approximately 21.2B total parameters and 3.5B active parameters. Built on SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts text backbone, it integrates an AuT speech encoder and a Mimi-style neural audio codec independently trained by KRAFTON AI, unifying speech understanding and generation in a single multimodal model. It is trained with the recipe of Raon-Speech-9B, KRAFTON's open-source speech language model.

Key Features

• End-to-End Speech Language Model: 21.2B-parameter (approximately 3.5B active) multimodal model built on SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts backbone.
• Independently Trained Speech Modules: AuT speech encoder (approximately 317M parameters) and Mimi-style neural audio codec (approximately 96M parameters), trained on curated data with an in-house pipeline.
• Bilingual Support: First in Korean and third in English aggregate score among publicly available speech language models at or below the 30B scale.
• Multi-Task Capabilities: STT, TTS, SpeechQA, SpokenQA, and turn-based multimodal chat in a single unified model.
• Paralinguistic Awareness: Uses emotion, intonation, and other information carried in the user's voice to generate natural spoken responses.
• Speaker Voice Conditioning: TTS with optional speaker reference audio via ECAPA embeddings.
• TTS Continuation: Generates speech that naturally continues from a reference audio with prefill-based continuation for seamless prosody.

Architecture

Component	Configuration
Text backbone	A.X K2 Light 20B-A3B MoE, 48 layers, 2,048 hidden size
Experts	128 routed experts, top-8 routing
Speech encoder	Independently trained AuT encoder based on the Qwen3-ASR architecture, 24 layers, approximately 317M parameters
Speech codec	Independently trained Mimi-style codec, approximately 96M parameters
Codec configuration	32 quantizers, 2,048-entry codebooks, 24 kHz, 12.5 frames per second
SpeechLM code interface	8 input code groups and 16 generated output code groups
Talker	Qwen3-based talker, 4 layers, 2,048 hidden size
Text-backbone context window	Up to 131,072 tokens

All components for the core speech-understanding and direct-TTS paths are included in the checkpoint. Speaker-conditioned and continuation TTS additionally use a frozen SpeechBrain ECAPA speaker encoder, downloaded automatically on first use.

Evaluation

Evaluation covered six task areas — speech recognition, speech synthesis, speech understanding, spoken question answering, text question answering, and tool calling — using 46 benchmarks (24 Korean, 22 English), including LibriSpeech, KsponSpeech, VoiceBench, KVoiceBench, MMAU, KMMAU, API-Bank, and FunctionChat-Bench. Each result is normalized to the 0–1 range and aggregated per language; see the KRAFTON AI tech blog for the full methodology and results.

Among publicly available speech language models at or below the 30B scale, A.X K2 Raon-Speech ranked first on the Korean aggregate score (0.72) and third on the English aggregate score (0.75).

Per-category performance, normalized from zero to the highest score in each category. Comparison models are the top English performers below that support both English and Korean.

Aggregate scores

Korean Rank	Model	Korean	English	English Rank
1	A.X K2 Raon-Speech	0.72	0.75	3
2	Raon-Speech	0.70	0.79	1
3	Qwen3-Omni	0.65	0.78	2
4	HyperCLOVA X 8B Omni	0.60	0.68	8
5	MOSS-Audio	0.58	0.62	13
6	Qwen2.5-Omni	0.55	0.73	5
7	Fun-Audio-Chat	0.53	0.69	7
8	Step-Audio 2 mini	0.32	0.67	10
9	Interactive-Omni	0.27	0.71	6
10	Ming-Lite-Omni v1.5	0.21	0.68	8
10	AF-Next	0.21	0.65	11
12	COVO-Audio	0.18	0.41	15
—	MiniCPM-o 4.5	—	0.74	4
—	Kimi-Audio	—	0.65	11
—	Audio Flamingo 3	—	0.60	14

Speech encoder

The AuT speech encoder matches Qwen3-ASR AuT on SpeechLM subtasks while trained on approximately 1M hours of audio versus the 40M hours reported for Qwen3-ASR-1.7B AuT (values relative to Qwen AuT = 100%).

Task / Metric	English	Korean	Language Average
Speech Recognition	96.7%	104.6%	100.7%
SpokenQA (Accuracy)	90.3%	104.8%	97.6%
SpeechQA (Accuracy)	98.8%	100.2%	99.5%
Overall Language Average	95.3%	103.2%	99.2%

Speech codec

The codec approaches Mimi's performance (values relative to Mimi = 100%) while trained on approximately 500K hours of audio; Moshi reports a 7M-hour unsupervised corpus for audio pretraining.

Metric	English	Korean	Language Average
PESQ	83.7%	83.3%	83.5%
ESTOI	96.8%	94.7%	95.8%
Speech Recognition	100.7%	112.6%	106.7%
Speaker Similarity	96.9%	99.6%	98.3%
UTMOS	94.4%	95.7%	95.1%
Overall Language Average	94.5%	97.2%	95.9%

Post-training ablations

Against the immediately preceding checkpoint, tool-use fine-tuning improved the tool-call score from 0.543 to 0.740, paralinguistic self-distillation improved the emotion/speaker score from 0.566 to 0.613, and TTS direct preference optimization reduced the average TTS error rate from 3.6% to 3.1%.

Requirements

pip install -r requirements.txt

# Optional: lower-latency attention on supported CUDA systems
pip install flash-attn --no-build-isolation

Requires transformers>=4.57.1. speechbrain is required for the speaker-conditioned and continuation TTS APIs.

Load the model

from transformers import AutoConfig
from transformers.dynamic_module_utils import get_class_from_dynamic_module

MODEL_ID = "KRAFTON/A.X-K2-Raon-Speech-21B-A3B"

config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
RaonPipeline = get_class_from_dynamic_module(
    "modeling_raon.RaonPipeline",
    MODEL_ID,
    revision=getattr(config, "_commit_hash", None),
)

pipe = RaonPipeline(
    MODEL_ID,
    device="cuda",
    dtype="bfloat16",
)

The BF16 checkpoint holds approximately 42.4 GB of weights; an 80 GB GPU or a suitable multi-GPU configuration is recommended.

Usage

Speech-to-text

text = pipe.stt("audio.wav")

Text-to-speech

audio, sample_rate = pipe.tts("Hello, how are you?")
pipe.save_audio((audio, sample_rate), "output.wav")

Speaker-conditioned TTS

audio, sample_rate = pipe.tts(
    "안녕하세요. 만나서 반갑습니다.",
    speaker_audio="speaker_reference.wav",
)
pipe.save_audio((audio, sample_rate), "conditioned.wav")

TTS continuation from reference audio

audio, sample_rate = pipe.tts_continuation(
    "And this is how the story continues.",
    ref_audio="reference.wav",
)
pipe.save_audio((audio, sample_rate), "continuation.wav")

SpeechQA (audio context, text question)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "context.wav"},
            {"type": "text", "text": "Please answer the question in this audio."},
        ],
    }
]

response = pipe.chat(messages)

SpokenQA (audio context, audio question)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "spoken_question.wav"},
        ],
    }
]

response = pipe.chat(messages)

Training

The released checkpoint is the final model following:

1. Speech-module alignment with the A.X K2 Light backbone frozen
2. Joint multimodal supervised fine-tuning with text-task mixing
3. Text and cross-modal distillation from A.X K2 Light
4. Speech-conditioned tool-use fine-tuning
5. ParaBridge-style on-policy self-distillation for paralinguistic behavior
6. TTS direct preference optimization, targeting elongated or repetitive syllables, incomplete termination, and recognition errors in generated speech

Repository packaging and the inference APIs were validated end-to-end, including a 12-call smoke matrix covering English and Korean STT, direct TTS, speaker-conditioned TTS, continuation TTS, SpeechQA, and SpokenQA:

python validate_repo.py
python validate_model.py

Intended use

This checkpoint is intended for research and non-commercial prototyping in bilingual speech recognition and generation, spoken question answering, audio understanding, and speech-enabled assistants. Users are responsible for obtaining consent before voice conditioning or generation involving identifiable speakers.

Acknowledgments

This work was supported by the Ministry of Science and ICT (MSIT), Republic of Korea, through the National IT Industry Promotion Agency (NIPA) (Grant No. PJT-26-010018). This research was also conducted as part of the Sovereign AI Foundation Model Project (Data Track), organized by the Ministry of Science and ICT (MSIT) and supported by the National Information Society Agency (NIA), Republic of Korea (Grant No. 2026-AIData-WII01).

License

This model is released under the CC BY-NC 4.0 license. The bundled A.X K2 Light implementation code retains its Apache License 2.0 notices, which apply to that code only.