Any-to-Any
Transformers
Safetensors
English
Korean
raon
feature-extraction
speech
audio
multimodal
mixture-of-experts
text-to-speech
automatic-speech-recognition
custom_code
Instructions to use KRAFTON/A.X-K2-Raon-Speech-21B-A3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KRAFTON/A.X-K2-Raon-Speech-21B-A3B with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("KRAFTON/A.X-K2-Raon-Speech-21B-A3B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs: add full vLLM-Omni capability examples
Browse filesExpand the English and Korean deployment guides with capability mappings and examples for speaker-conditioned TTS, TextQA, SpeechQA, multimodal chat, and SpokenQA.
- README.md +117 -0
- README_ko.md +118 -0
README.md
CHANGED
|
@@ -275,6 +275,16 @@ compatible `feat/axk2_raon_speech` branch. The validated revision is
|
|
| 275 |
you can optionally pin it to match the vLLM-Omni implementation revision used
|
| 276 |
for validation.
|
| 277 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 278 |
### Install and serve
|
| 279 |
|
| 280 |
```bash
|
|
@@ -330,6 +340,26 @@ curl -fS -X POST http://localhost:8091/v1/audio/speech \
|
|
| 330 |
EOF
|
| 331 |
```
|
| 332 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 333 |
### STT
|
| 334 |
|
| 335 |
Encode the input audio as a portable single-line base64 data URL:
|
|
@@ -363,6 +393,93 @@ curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
|
| 363 |
EOF
|
| 364 |
```
|
| 365 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 366 |
## Training
|
| 367 |
|
| 368 |
The released checkpoint is the final model following:
|
|
|
|
| 275 |
you can optionally pin it to match the vLLM-Omni implementation revision used
|
| 276 |
for validation.
|
| 277 |
|
| 278 |
+
The online server supports these request patterns:
|
| 279 |
+
|
| 280 |
+
| Capability | Request | Response |
|
| 281 |
+
| --- | --- | --- |
|
| 282 |
+
| Direct or speaker-conditioned TTS / voice cloning | Text, optionally with reference audio | WAV from `/v1/audio/speech` |
|
| 283 |
+
| STT | Audio with a written transcription instruction | Text from `/v1/chat/completions` |
|
| 284 |
+
| TextQA | Written question | Text answer from `/v1/chat/completions` |
|
| 285 |
+
| SpeechQA / multimodal chat | Audio context with a written question or instruction | Text answer from `/v1/chat/completions` |
|
| 286 |
+
| SpokenQA / SpeechChat | Spoken audio question without a written prompt | Text answer from `/v1/chat/completions` |
|
| 287 |
+
|
| 288 |
### Install and serve
|
| 289 |
|
| 290 |
```bash
|
|
|
|
| 340 |
EOF
|
| 341 |
```
|
| 342 |
|
| 343 |
+
### Speaker-conditioned TTS / voice cloning
|
| 344 |
+
|
| 345 |
+
Encode the reference speaker audio as a portable single-line base64 data URL:
|
| 346 |
+
|
| 347 |
+
```bash
|
| 348 |
+
REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"
|
| 349 |
+
|
| 350 |
+
curl -fS -X POST http://localhost:8091/v1/audio/speech \
|
| 351 |
+
-H "Content-Type: application/json" \
|
| 352 |
+
--data-binary @- \
|
| 353 |
+
--output axk2_conditioned.wav <<EOF
|
| 354 |
+
{
|
| 355 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 356 |
+
"input": "Hello, this sentence uses the reference speaker.",
|
| 357 |
+
"ref_audio": "data:audio/wav;base64,$REF_AUDIO_B64",
|
| 358 |
+
"response_format": "wav"
|
| 359 |
+
}
|
| 360 |
+
EOF
|
| 361 |
+
```
|
| 362 |
+
|
| 363 |
### STT
|
| 364 |
|
| 365 |
Encode the input audio as a portable single-line base64 data URL:
|
|
|
|
| 393 |
EOF
|
| 394 |
```
|
| 395 |
|
| 396 |
+
### TextQA
|
| 397 |
+
|
| 398 |
+
Send a text-only question:
|
| 399 |
+
|
| 400 |
+
```bash
|
| 401 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 402 |
+
-H "Content-Type: application/json" \
|
| 403 |
+
--data-binary @- <<'EOF'
|
| 404 |
+
{
|
| 405 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 406 |
+
"messages": [
|
| 407 |
+
{
|
| 408 |
+
"role": "user",
|
| 409 |
+
"content": [
|
| 410 |
+
{
|
| 411 |
+
"type": "text",
|
| 412 |
+
"text": "What is the capital of South Korea?"
|
| 413 |
+
}
|
| 414 |
+
]
|
| 415 |
+
}
|
| 416 |
+
]
|
| 417 |
+
}
|
| 418 |
+
EOF
|
| 419 |
+
```
|
| 420 |
+
|
| 421 |
+
### SpeechQA / multimodal chat
|
| 422 |
+
|
| 423 |
+
Use the audio as context and include a written question:
|
| 424 |
+
|
| 425 |
+
```bash
|
| 426 |
+
AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"
|
| 427 |
+
|
| 428 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 429 |
+
-H "Content-Type: application/json" \
|
| 430 |
+
--data-binary @- <<EOF
|
| 431 |
+
{
|
| 432 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 433 |
+
"messages": [
|
| 434 |
+
{
|
| 435 |
+
"role": "user",
|
| 436 |
+
"content": [
|
| 437 |
+
{
|
| 438 |
+
"type": "audio_url",
|
| 439 |
+
"audio_url": {
|
| 440 |
+
"url": "data:audio/wav;base64,$AUDIO_B64"
|
| 441 |
+
}
|
| 442 |
+
},
|
| 443 |
+
{
|
| 444 |
+
"type": "text",
|
| 445 |
+
"text": "What is the speaker talking about?"
|
| 446 |
+
}
|
| 447 |
+
]
|
| 448 |
+
}
|
| 449 |
+
]
|
| 450 |
+
}
|
| 451 |
+
EOF
|
| 452 |
+
```
|
| 453 |
+
|
| 454 |
+
### SpokenQA / SpeechChat
|
| 455 |
+
|
| 456 |
+
Send a spoken question without a written prompt:
|
| 457 |
+
|
| 458 |
+
```bash
|
| 459 |
+
QUESTION_AUDIO_B64="$(base64 < question.wav | tr -d '\n')"
|
| 460 |
+
|
| 461 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 462 |
+
-H "Content-Type: application/json" \
|
| 463 |
+
--data-binary @- <<EOF
|
| 464 |
+
{
|
| 465 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 466 |
+
"messages": [
|
| 467 |
+
{
|
| 468 |
+
"role": "user",
|
| 469 |
+
"content": [
|
| 470 |
+
{
|
| 471 |
+
"type": "audio_url",
|
| 472 |
+
"audio_url": {
|
| 473 |
+
"url": "data:audio/wav;base64,$QUESTION_AUDIO_B64"
|
| 474 |
+
}
|
| 475 |
+
}
|
| 476 |
+
]
|
| 477 |
+
}
|
| 478 |
+
]
|
| 479 |
+
}
|
| 480 |
+
EOF
|
| 481 |
+
```
|
| 482 |
+
|
| 483 |
## Training
|
| 484 |
|
| 485 |
The released checkpoint is the final model following:
|
README_ko.md
CHANGED
|
@@ -275,6 +275,16 @@ KRAFTON의 공식
|
|
| 275 |
[`7bcaf21`](https://github.com/krafton-ai/vllm-omni/commit/7bcaf21df881bda4280cbbf9752fc356e753d3bb)이며,
|
| 276 |
검증에 사용된 vLLM-Omni 구현 리비전과 일치시키려면 이 커밋으로 고정하세요.
|
| 277 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 278 |
### 설치 및 실행
|
| 279 |
|
| 280 |
```bash
|
|
@@ -329,6 +339,27 @@ curl -fS -X POST http://localhost:8091/v1/audio/speech \
|
|
| 329 |
EOF
|
| 330 |
```
|
| 331 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 332 |
### STT
|
| 333 |
|
| 334 |
입력 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로 인코딩합니다.
|
|
@@ -362,6 +393,93 @@ curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
|
| 362 |
EOF
|
| 363 |
```
|
| 364 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 365 |
## Training
|
| 366 |
|
| 367 |
공개된 checkpoint는 다음 과정을 거친 최종 모델입니다.
|
|
|
|
| 275 |
[`7bcaf21`](https://github.com/krafton-ai/vllm-omni/commit/7bcaf21df881bda4280cbbf9752fc356e753d3bb)이며,
|
| 276 |
검증에 사용된 vLLM-Omni 구현 리비전과 일치시키려면 이 커밋으로 고정하세요.
|
| 277 |
|
| 278 |
+
온라인 서버는 다음 요청 형식을 지원합니다.
|
| 279 |
+
|
| 280 |
+
| 기능 | 요청 | 응답 |
|
| 281 |
+
| --- | --- | --- |
|
| 282 |
+
| 기본 또는 화자 조건 TTS / 음성 복제 | 텍스트와 선택적 참조 오디오 | `/v1/audio/speech`에서 WAV |
|
| 283 |
+
| STT | 오디오와 텍스트 전사 지시문 | `/v1/chat/completions`에서 텍스트 |
|
| 284 |
+
| TextQA | 텍스트 질문 | `/v1/chat/completions`에서 텍스트 답변 |
|
| 285 |
+
| SpeechQA / 멀티모달 대화 | 오디오 맥락과 텍스트 질문 또는 지시문 | `/v1/chat/completions`에서 텍스트 답변 |
|
| 286 |
+
| SpokenQA / SpeechChat | 텍스트 프롬프트 없이 음성으로 전달한 질문 | `/v1/chat/completions`에서 텍스트 답변 |
|
| 287 |
+
|
| 288 |
### 설치 및 실행
|
| 289 |
|
| 290 |
```bash
|
|
|
|
| 339 |
EOF
|
| 340 |
```
|
| 341 |
|
| 342 |
+
### 화자 조건 TTS / 음성 복제
|
| 343 |
+
|
| 344 |
+
참조 화자의 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로
|
| 345 |
+
인코딩합니다.
|
| 346 |
+
|
| 347 |
+
```bash
|
| 348 |
+
REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"
|
| 349 |
+
|
| 350 |
+
curl -fS -X POST http://localhost:8091/v1/audio/speech \
|
| 351 |
+
-H "Content-Type: application/json" \
|
| 352 |
+
--data-binary @- \
|
| 353 |
+
--output axk2_conditioned.wav <<EOF
|
| 354 |
+
{
|
| 355 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 356 |
+
"input": "Hello, this sentence uses the reference speaker.",
|
| 357 |
+
"ref_audio": "data:audio/wav;base64,$REF_AUDIO_B64",
|
| 358 |
+
"response_format": "wav"
|
| 359 |
+
}
|
| 360 |
+
EOF
|
| 361 |
+
```
|
| 362 |
+
|
| 363 |
### STT
|
| 364 |
|
| 365 |
입력 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로 인코딩합니다.
|
|
|
|
| 393 |
EOF
|
| 394 |
```
|
| 395 |
|
| 396 |
+
### TextQA
|
| 397 |
+
|
| 398 |
+
텍스트로만 구성된 질문을 전송합니다.
|
| 399 |
+
|
| 400 |
+
```bash
|
| 401 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 402 |
+
-H "Content-Type: application/json" \
|
| 403 |
+
--data-binary @- <<'EOF'
|
| 404 |
+
{
|
| 405 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 406 |
+
"messages": [
|
| 407 |
+
{
|
| 408 |
+
"role": "user",
|
| 409 |
+
"content": [
|
| 410 |
+
{
|
| 411 |
+
"type": "text",
|
| 412 |
+
"text": "What is the capital of South Korea?"
|
| 413 |
+
}
|
| 414 |
+
]
|
| 415 |
+
}
|
| 416 |
+
]
|
| 417 |
+
}
|
| 418 |
+
EOF
|
| 419 |
+
```
|
| 420 |
+
|
| 421 |
+
### SpeechQA / 멀티모달 대화
|
| 422 |
+
|
| 423 |
+
오디오를 맥락으로 사용하고 텍스트 질문을 함께 전송합니다.
|
| 424 |
+
|
| 425 |
+
```bash
|
| 426 |
+
AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"
|
| 427 |
+
|
| 428 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 429 |
+
-H "Content-Type: application/json" \
|
| 430 |
+
--data-binary @- <<EOF
|
| 431 |
+
{
|
| 432 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 433 |
+
"messages": [
|
| 434 |
+
{
|
| 435 |
+
"role": "user",
|
| 436 |
+
"content": [
|
| 437 |
+
{
|
| 438 |
+
"type": "audio_url",
|
| 439 |
+
"audio_url": {
|
| 440 |
+
"url": "data:audio/wav;base64,$AUDIO_B64"
|
| 441 |
+
}
|
| 442 |
+
},
|
| 443 |
+
{
|
| 444 |
+
"type": "text",
|
| 445 |
+
"text": "What is the speaker talking about?"
|
| 446 |
+
}
|
| 447 |
+
]
|
| 448 |
+
}
|
| 449 |
+
]
|
| 450 |
+
}
|
| 451 |
+
EOF
|
| 452 |
+
```
|
| 453 |
+
|
| 454 |
+
### SpokenQA / SpeechChat
|
| 455 |
+
|
| 456 |
+
텍스트 프롬프트 없이 음성 질문을 전송합니다.
|
| 457 |
+
|
| 458 |
+
```bash
|
| 459 |
+
QUESTION_AUDIO_B64="$(base64 < question.wav | tr -d '\n')"
|
| 460 |
+
|
| 461 |
+
curl -fS -X POST http://localhost:8091/v1/chat/completions \
|
| 462 |
+
-H "Content-Type: application/json" \
|
| 463 |
+
--data-binary @- <<EOF
|
| 464 |
+
{
|
| 465 |
+
"model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
|
| 466 |
+
"messages": [
|
| 467 |
+
{
|
| 468 |
+
"role": "user",
|
| 469 |
+
"content": [
|
| 470 |
+
{
|
| 471 |
+
"type": "audio_url",
|
| 472 |
+
"audio_url": {
|
| 473 |
+
"url": "data:audio/wav;base64,$QUESTION_AUDIO_B64"
|
| 474 |
+
}
|
| 475 |
+
}
|
| 476 |
+
]
|
| 477 |
+
}
|
| 478 |
+
]
|
| 479 |
+
}
|
| 480 |
+
EOF
|
| 481 |
+
```
|
| 482 |
+
|
| 483 |
## Training
|
| 484 |
|
| 485 |
공개된 checkpoint는 다음 과정을 거친 최종 모델입니다.
|