ddwkim commited on
Commit
3e2c73e
·
verified ·
1 Parent(s): 3d56cdc

docs: add full vLLM-Omni capability examples

Browse files

Expand the English and Korean deployment guides with capability mappings and examples for speaker-conditioned TTS, TextQA, SpeechQA, multimodal chat, and SpokenQA.

Files changed (2) hide show
  1. README.md +117 -0
  2. README_ko.md +118 -0
README.md CHANGED
@@ -275,6 +275,16 @@ compatible `feat/axk2_raon_speech` branch. The validated revision is
275
  you can optionally pin it to match the vLLM-Omni implementation revision used
276
  for validation.
277
 
 
 
 
 
 
 
 
 
 
 
278
  ### Install and serve
279
 
280
  ```bash
@@ -330,6 +340,26 @@ curl -fS -X POST http://localhost:8091/v1/audio/speech \
330
  EOF
331
  ```
332
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
333
  ### STT
334
 
335
  Encode the input audio as a portable single-line base64 data URL:
@@ -363,6 +393,93 @@ curl -fS -X POST http://localhost:8091/v1/chat/completions \
363
  EOF
364
  ```
365
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
366
  ## Training
367
 
368
  The released checkpoint is the final model following:
 
275
  you can optionally pin it to match the vLLM-Omni implementation revision used
276
  for validation.
277
 
278
+ The online server supports these request patterns:
279
+
280
+ | Capability | Request | Response |
281
+ | --- | --- | --- |
282
+ | Direct or speaker-conditioned TTS / voice cloning | Text, optionally with reference audio | WAV from `/v1/audio/speech` |
283
+ | STT | Audio with a written transcription instruction | Text from `/v1/chat/completions` |
284
+ | TextQA | Written question | Text answer from `/v1/chat/completions` |
285
+ | SpeechQA / multimodal chat | Audio context with a written question or instruction | Text answer from `/v1/chat/completions` |
286
+ | SpokenQA / SpeechChat | Spoken audio question without a written prompt | Text answer from `/v1/chat/completions` |
287
+
288
  ### Install and serve
289
 
290
  ```bash
 
340
  EOF
341
  ```
342
 
343
+ ### Speaker-conditioned TTS / voice cloning
344
+
345
+ Encode the reference speaker audio as a portable single-line base64 data URL:
346
+
347
+ ```bash
348
+ REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"
349
+
350
+ curl -fS -X POST http://localhost:8091/v1/audio/speech \
351
+ -H "Content-Type: application/json" \
352
+ --data-binary @- \
353
+ --output axk2_conditioned.wav <<EOF
354
+ {
355
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
356
+ "input": "Hello, this sentence uses the reference speaker.",
357
+ "ref_audio": "data:audio/wav;base64,$REF_AUDIO_B64",
358
+ "response_format": "wav"
359
+ }
360
+ EOF
361
+ ```
362
+
363
  ### STT
364
 
365
  Encode the input audio as a portable single-line base64 data URL:
 
393
  EOF
394
  ```
395
 
396
+ ### TextQA
397
+
398
+ Send a text-only question:
399
+
400
+ ```bash
401
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
402
+ -H "Content-Type: application/json" \
403
+ --data-binary @- <<'EOF'
404
+ {
405
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
406
+ "messages": [
407
+ {
408
+ "role": "user",
409
+ "content": [
410
+ {
411
+ "type": "text",
412
+ "text": "What is the capital of South Korea?"
413
+ }
414
+ ]
415
+ }
416
+ ]
417
+ }
418
+ EOF
419
+ ```
420
+
421
+ ### SpeechQA / multimodal chat
422
+
423
+ Use the audio as context and include a written question:
424
+
425
+ ```bash
426
+ AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"
427
+
428
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
429
+ -H "Content-Type: application/json" \
430
+ --data-binary @- <<EOF
431
+ {
432
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
433
+ "messages": [
434
+ {
435
+ "role": "user",
436
+ "content": [
437
+ {
438
+ "type": "audio_url",
439
+ "audio_url": {
440
+ "url": "data:audio/wav;base64,$AUDIO_B64"
441
+ }
442
+ },
443
+ {
444
+ "type": "text",
445
+ "text": "What is the speaker talking about?"
446
+ }
447
+ ]
448
+ }
449
+ ]
450
+ }
451
+ EOF
452
+ ```
453
+
454
+ ### SpokenQA / SpeechChat
455
+
456
+ Send a spoken question without a written prompt:
457
+
458
+ ```bash
459
+ QUESTION_AUDIO_B64="$(base64 < question.wav | tr -d '\n')"
460
+
461
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
462
+ -H "Content-Type: application/json" \
463
+ --data-binary @- <<EOF
464
+ {
465
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
466
+ "messages": [
467
+ {
468
+ "role": "user",
469
+ "content": [
470
+ {
471
+ "type": "audio_url",
472
+ "audio_url": {
473
+ "url": "data:audio/wav;base64,$QUESTION_AUDIO_B64"
474
+ }
475
+ }
476
+ ]
477
+ }
478
+ ]
479
+ }
480
+ EOF
481
+ ```
482
+
483
  ## Training
484
 
485
  The released checkpoint is the final model following:
README_ko.md CHANGED
@@ -275,6 +275,16 @@ KRAFTON의 공식
275
  [`7bcaf21`](https://github.com/krafton-ai/vllm-omni/commit/7bcaf21df881bda4280cbbf9752fc356e753d3bb)이며,
276
  검증에 사용된 vLLM-Omni 구현 리비전과 일치시키려면 이 커밋으로 고정하세요.
277
 
 
 
 
 
 
 
 
 
 
 
278
  ### 설치 및 실행
279
 
280
  ```bash
@@ -329,6 +339,27 @@ curl -fS -X POST http://localhost:8091/v1/audio/speech \
329
  EOF
330
  ```
331
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
332
  ### STT
333
 
334
  입력 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로 인코딩합니다.
@@ -362,6 +393,93 @@ curl -fS -X POST http://localhost:8091/v1/chat/completions \
362
  EOF
363
  ```
364
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
365
  ## Training
366
 
367
  공개된 checkpoint는 다음 과정을 거친 최종 모델입니다.
 
275
  [`7bcaf21`](https://github.com/krafton-ai/vllm-omni/commit/7bcaf21df881bda4280cbbf9752fc356e753d3bb)이며,
276
  검증에 사용된 vLLM-Omni 구현 리비전과 일치시키려면 이 커밋으로 고정하세요.
277
 
278
+ 온라인 서버는 다음 요청 형식을 지원합니다.
279
+
280
+ | 기능 | 요청 | 응답 |
281
+ | --- | --- | --- |
282
+ | 기본 또는 화자 조건 TTS / 음성 복제 | 텍스트와 선택적 참조 오디오 | `/v1/audio/speech`에서 WAV |
283
+ | STT | 오디오와 텍스트 전사 지시문 | `/v1/chat/completions`에서 텍스트 |
284
+ | TextQA | 텍스트 질문 | `/v1/chat/completions`에서 텍스트 답변 |
285
+ | SpeechQA / 멀티모달 대화 | 오디오 맥락과 텍스트 질문 또는 지시문 | `/v1/chat/completions`에서 텍스트 답변 |
286
+ | SpokenQA / SpeechChat | 텍스트 프롬프트 없이 음성으로 전달한 질문 | `/v1/chat/completions`에서 텍스트 답변 |
287
+
288
  ### 설치 및 실행
289
 
290
  ```bash
 
339
  EOF
340
  ```
341
 
342
+ ### 화자 조건 TTS / 음성 복제
343
+
344
+ 참조 화자의 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로
345
+ 인코딩합니다.
346
+
347
+ ```bash
348
+ REF_AUDIO_B64="$(base64 < speaker_ref.wav | tr -d '\n')"
349
+
350
+ curl -fS -X POST http://localhost:8091/v1/audio/speech \
351
+ -H "Content-Type: application/json" \
352
+ --data-binary @- \
353
+ --output axk2_conditioned.wav <<EOF
354
+ {
355
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
356
+ "input": "Hello, this sentence uses the reference speaker.",
357
+ "ref_audio": "data:audio/wav;base64,$REF_AUDIO_B64",
358
+ "response_format": "wav"
359
+ }
360
+ EOF
361
+ ```
362
+
363
  ### STT
364
 
365
  입력 오디오를 이식성이 높은 한 줄짜리 base64 데이터 URL로 인코딩합니다.
 
393
  EOF
394
  ```
395
 
396
+ ### TextQA
397
+
398
+ 텍스트로만 구성된 질문을 전송합니다.
399
+
400
+ ```bash
401
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
402
+ -H "Content-Type: application/json" \
403
+ --data-binary @- <<'EOF'
404
+ {
405
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
406
+ "messages": [
407
+ {
408
+ "role": "user",
409
+ "content": [
410
+ {
411
+ "type": "text",
412
+ "text": "What is the capital of South Korea?"
413
+ }
414
+ ]
415
+ }
416
+ ]
417
+ }
418
+ EOF
419
+ ```
420
+
421
+ ### SpeechQA / 멀티모달 대화
422
+
423
+ 오디오를 맥락으로 사용하고 텍스트 질문을 함께 전송합니다.
424
+
425
+ ```bash
426
+ AUDIO_B64="$(base64 < audio.wav | tr -d '\n')"
427
+
428
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
429
+ -H "Content-Type: application/json" \
430
+ --data-binary @- <<EOF
431
+ {
432
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
433
+ "messages": [
434
+ {
435
+ "role": "user",
436
+ "content": [
437
+ {
438
+ "type": "audio_url",
439
+ "audio_url": {
440
+ "url": "data:audio/wav;base64,$AUDIO_B64"
441
+ }
442
+ },
443
+ {
444
+ "type": "text",
445
+ "text": "What is the speaker talking about?"
446
+ }
447
+ ]
448
+ }
449
+ ]
450
+ }
451
+ EOF
452
+ ```
453
+
454
+ ### SpokenQA / SpeechChat
455
+
456
+ 텍스트 프롬프트 없이 음성 질문을 전송합니다.
457
+
458
+ ```bash
459
+ QUESTION_AUDIO_B64="$(base64 < question.wav | tr -d '\n')"
460
+
461
+ curl -fS -X POST http://localhost:8091/v1/chat/completions \
462
+ -H "Content-Type: application/json" \
463
+ --data-binary @- <<EOF
464
+ {
465
+ "model": "KRAFTON/A.X-K2-Raon-Speech-21B-A3B",
466
+ "messages": [
467
+ {
468
+ "role": "user",
469
+ "content": [
470
+ {
471
+ "type": "audio_url",
472
+ "audio_url": {
473
+ "url": "data:audio/wav;base64,$QUESTION_AUDIO_B64"
474
+ }
475
+ }
476
+ ]
477
+ }
478
+ ]
479
+ }
480
+ EOF
481
+ ```
482
+
483
  ## Training
484
 
485
  공개된 checkpoint는 다음 과정을 거친 최종 모델입니다.