KRAFTON/A.X-K2-Raon-Speech-21B-A3B

A.X K2 Raon-Speech

English | 한국어

A.X K2 Raon-Speech is a bilingual English/Korean speech language model with approximately 21.2B total parameters and 3.5B active parameters. Built on SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts text backbone, it integrates an AuT speech encoder and a Mimi-style neural audio codec independently trained by KRAFTON AI, unifying speech understanding and generation in a single multimodal model. It is trained with the recipe of Raon-Speech-9B, KRAFTON's open-source speech language model.

Key Features

• End-to-End Speech Language Model: 21.2B-parameter (approximately 3.5B active) multimodal model built on SK Telecom's A.X K2 Light 20B-A3B mixture-of-experts backbone.
• Independently Trained Speech Modules: AuT speech encoder (approximately 317M parameters) and Mimi-style neural audio codec (approximately 96M parameters), trained on curated data with an in-house pipeline.
• Bilingual Support: First in Korean and third in English aggregate score among publicly available speech language models at or below the 30B scale.
• Multi-Task Capabilities: STT, TTS, SpeechQA, SpokenQA, and turn-based multimodal chat in a single unified model.
• Paralinguistic Awareness: Uses emotion, intonation, and other information carried in the user's voice to generate natural spoken responses.
• Speaker Voice Conditioning: TTS with optional speaker reference audio via ECAPA embeddings.
• TTS Continuation: Generates speech that naturally continues from a reference audio with prefill-based continuation for seamless prosody.

Architecture

Component	Configuration
Text backbone	A.X K2 Light 20B-A3B MoE, 48 layers, 2,048 hidden size
Experts	128 routed experts, top-8 routing
Speech encoder	Independently trained AuT encoder based on the Qwen3-ASR architecture, 24 layers, approximately 317M parameters
Speech codec	Independently trained Mimi-style codec, approximately 96M parameters
Codec configuration	32 quantizers, 2,048-entry codebooks, 24 kHz, 12.5 frames per second
SpeechLM code interface	8 input code groups and 16 generated output code groups
Talker	Qwen3-based talker, 4 layers, 2,048 hidden size
Text-backbone context window	Up to 131,072 tokens

All components for the core speech-understanding and direct-TTS paths are included in the checkpoint. Speaker-conditioned and continuation TTS additionally use a frozen SpeechBrain ECAPA speaker encoder, downloaded automatically on first use.

Evaluation

Evaluation covered six task areas — speech recognition, speech synthesis, speech understanding, spoken question answering, text question answering, and tool calling — using 46 benchmarks (24 Korean, 22 English), including LibriSpeech, KsponSpeech, VoiceBench, KVoiceBench, MMAU, KMMAU, API-Bank, and FunctionChat-Bench. Each result is normalized to the 0–1 range and aggregated per language; see the KRAFTON AI tech blog for the full methodology and results.

Among publicly available speech language models at or below the 30B scale, A.X K2 Raon-Speech ranked first on the Korean aggregate score (0.72) and third on the English aggregate score (0.75).

Per-category performance, normalized from zero to the highest score in each category. Comparison models are the top English performers below that support both English and Korean.

Aggregate scores

Korean Rank	Model	Korean	English	English Rank
1	A.X K2 Raon-Speech	0.72	0.75	3
2	Raon-Speech	0.70	0.79	1
3	Qwen3-Omni	0.65	0.78	2
4	HyperCLOVA X 8B Omni	0.60	0.68	8
5	MOSS-Audio	0.58	0.62	13
6	Qwen2.5-Omni	0.55	0.73	5
7	Fun-Audio-Chat	0.53	0.69	7
8	Step-Audio 2 mini	0.32	0.67	10
9	Interactive-Omni	0.27	0.71	6
10	Ming-Lite-Omni v1.5	0.21	0.68	8
10	AF-Next	0.21	0.65	11
12	COVO-Audio	0.18	0.41	15
—	MiniCPM-o 4.5	—	0.74	4
—	Kimi-Audio	—	0.65	11
—	Audio Flamingo 3	—	0.60	14

Speech encoder

The AuT speech encoder matches Qwen3-ASR AuT on SpeechLM subtasks while trained on approximately 1M hours of audio versus the 40M hours reported for Qwen3-ASR-1.7B AuT (values relative to Qwen AuT = 100%).

Task / Metric	English	Korean	Language Average
Speech Recognition	96.7%	104.6%	100.7%
SpokenQA (Accuracy)	90.3%	104.8%	97.6%
SpeechQA (Accuracy)	98.8%	100.2%	99.5%
Overall Language Average	95.3%	103.2%	99.2%

Speech codec

The codec approaches Mimi's performance (values relative to Mimi = 100%) while trained on approximately 500K hours of audio; Moshi reports a 7M-hour unsupervised corpus for audio pretraining.

Metric	English	Korean	Language Average
PESQ	83.7%	83.3%	83.5%
ESTOI	96.8%	94.7%	95.8%
Speech Recognition	100.7%	112.6%	106.7%
Speaker Similarity	96.9%	99.6%	98.3%
UTMOS	94.4%	95.7%	95.1%
Overall Language Average	94.5%	97.2%	95.9%

Post-training ablations

Against the immediately preceding checkpoint, tool-use fine-tuning improved the tool-call score from 0.543 to 0.740, paralinguistic self-distillation improved the emotion/speaker score from 0.566 to 0.613, and TTS direct preference optimization reduced the average TTS error rate from 3.6% to 3.1%.

Requirements

pip install -r requirements.txt

# Optional: lower-latency attention on supported CUDA systems
pip install flash-attn --no-build-isolation

Requires transformers>=4.57.1. speechbrain is required for the speaker-conditioned and continuation TTS APIs.

Load the model

from transformers import AutoConfig
from transformers.dynamic_module_utils import get_class_from_dynamic_module

MODEL_ID = "KRAFTON/A.X-K2-Raon-Speech-21B-A3B"

config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
RaonPipeline = get_class_from_dynamic_module(
    "modeling_raon.RaonPipeline",
    MODEL_ID,
    revision=getattr(config, "_commit_hash", None),
)

pipe = RaonPipeline(
    MODEL_ID,
    device="cuda",
    dtype="bfloat16",
)

The BF16 checkpoint holds approximately 42.4 GB of weights; an 80 GB GPU or a suitable multi-GPU configuration is recommended.

Usage

Speech-to-text

text = pipe.stt("audio.wav")

Text-to-speech

audio, sample_rate = pipe.tts("Hello, how are you?")
pipe.save_audio((audio, sample_rate), "output.wav")

Speaker-conditioned TTS

audio, sample_rate = pipe.tts(
    "안녕하세요. 만나서 반갑습니다.",
    speaker_audio="speaker_reference.wav",
)
pipe.save_audio((audio, sample_rate), "conditioned.wav")

TTS continuation from reference audio

audio, sample_rate = pipe.tts_continuation(
    "And this is how the story continues.",
    ref_audio="reference.wav",
)
pipe.save_audio((audio, sample_rate), "continuation.wav")

SpeechQA (audio context, text question)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "context.wav"},
            {"type": "text", "text": "Please answer the question in this audio."},
        ],
    }
]

response = pipe.chat(messages)

SpokenQA (audio context, audio question)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "spoken_question.wav"},
        ],
    }
]

response = pipe.chat(messages)

Deployment with vLLM-Omni

Use KRAFTON's official vLLM-Omni repository and the compatible feat/axk2_raon_speech branch. The validated revision is 7bcaf21; you can optionally pin it to match the vLLM-Omni implementation revision used for validation.

The online server supports these request patterns:

Capability	Request	Response
Direct or speaker-conditioned TTS / voice cloning	Text, optionally with reference audio	WAV from /v1/audio/speech
STT	Audio with a written transcription instruction	Text from /v1/chat/completions
TextQA	Written question	Text answer from /v1/chat/completions
SpeechQA / multimodal chat	Audio context with a written question or instruction	Text answer from /v1/chat/completions
SpokenQA / SpeechChat	Spoken audio question without a written prompt	Text answer from /v1/chat/completions

Install and serve

git clone --branch feat/axk2_raon_speech --single-branch \
    https://github.com/krafton-ai/vllm-omni.git
cd vllm-omni

# Optional: pin the revision validated with this checkpoint.
git checkout 7bcaf21df881bda4280cbbf9752fc356e753d3bb

python -m pip install "vllm==0.20.0"
python -m pip install -e .

The bundled stage profile targets one GPU with approximately 96 GB of VRAM. It uses tensor parallelism 1 (TP=1) and sets both the maximum model length and maximum batched tokens to 4096. With vLLM 0.20, VLLM_MLA_DISABLE=1 is required because this checkpoint has a 192-wide KV-cache head.

From the cloned repository root, start the server with:

export VLLM_MLA_DISABLE=1

vllm-omni serve KRAFTON/A.X-K2-Raon-Speech-21B-A3B \
    --stage-configs-path examples/offline_inference/raon/raon_axk1_local.yaml \
    --host 0.0.0.0 \
    --port 8091 \
    --trust-remote-code \
    --omni

Health and model listing

curl -fS http://localhost:8091/health
curl -fS http://localhost:8091/v1/models

TTS

curl -fS -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    --data-binary @- \
    --output axk2_tts.wav <<'EOF'
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "input": "Hello, this is A.X-K2 Raon Speech.",
  "response_format": "wav"
}
EOF

Speaker-conditioned TTS / voice cloning

Encode the reference speaker audio as a portable single-line base64 data URL:

REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"

curl -fS -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    --data-binary @- \
    --output axk2_conditioned.wav <<EOF
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "input": "Hello, this sentence uses the reference speaker.",
  "ref_audio": "data:audio/wav;base64,$REF_AUDIO_B64",
  "response_format": "wav"
}
EOF

STT

Encode the input audio as a portable single-line base64 data URL:

AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"

curl -fS -X POST http://localhost:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    --data-binary @- <<EOF
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "audio_url",
          "audio_url": {
            "url": "data:audio/wav;base64,$AUDIO_B64"
          }
        },
        {
          "type": "text",
          "text": "Transcribe the audio into text."
        }
      ]
    }
  ]
}
EOF

TextQA

Send a text-only question:

curl -fS -X POST http://localhost:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    --data-binary @- <<'EOF'
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is the capital of South Korea?"
        }
      ]
    }
  ]
}
EOF

SpeechQA / multimodal chat

Use the audio as context and include a written question:

AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"

curl -fS -X POST http://localhost:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    --data-binary @- <<EOF
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "audio_url",
          "audio_url": {
            "url": "data:audio/wav;base64,$AUDIO_B64"
          }
        },
        {
          "type": "text",
          "text": "What is the speaker talking about?"
        }
      ]
    }
  ]
}
EOF

SpokenQA / SpeechChat

Send a spoken question without a written prompt:

QUESTION_AUDIO_B64="$(base64 < question.wav | tr -d '\n')"

curl -fS -X POST http://localhost:8091/v1/chat/completions \
    -H "Content-Type: application/json" \
    --data-binary @- <<EOF
{
  "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "audio_url",
          "audio_url": {
            "url": "data:audio/wav;base64,$QUESTION_AUDIO_B64"
          }
        }
      ]
    }
  ]
}
EOF

Training

The released checkpoint is the final model following:

1. Speech-module alignment with the A.X K2 Light backbone frozen
2. Joint multimodal supervised fine-tuning with text-task mixing
3. Text and cross-modal distillation from A.X K2 Light
4. Speech-conditioned tool-use fine-tuning
5. ParaBridge-style on-policy self-distillation for paralinguistic behavior
6. TTS direct preference optimization, targeting elongated or repetitive syllables, incomplete termination, and recognition errors in generated speech

Repository packaging and the inference APIs were validated end-to-end, including a 12-call smoke matrix covering English and Korean STT, direct TTS, speaker-conditioned TTS, continuation TTS, SpeechQA, and SpokenQA:

python validate_repo.py
python validate_model.py

Intended use

This checkpoint is intended for research and non-commercial prototyping in bilingual speech recognition and generation, spoken question answering, audio understanding, and speech-enabled assistants. Users are responsible for obtaining consent before voice conditioning or generation involving identifiable speakers.

Acknowledgments

This work was supported by the Ministry of Science and ICT (MSIT), Republic of Korea, through the National IT Industry Promotion Agency (NIPA) (Grant No. PJT-26-010018). This research was also conducted as part of the Sovereign AI Foundation Model Project (Data Track), organized by the Ministry of Science and ICT (MSIT) and supported by the National Information Society Agency (NIA), Republic of Korea (Grant No. 2026-AIData-WII01).

License

This model is released under the CC BY-NC 4.0 license. The bundled A.X K2 Light implementation code retains its Apache License 2.0 notices, which apply to that code only.