Instructions to use OpenMOSS-Team/MOSS-Transcribe-Diarize with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMOSS-Team/MOSS-Transcribe-Diarize with Transformers:
# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("OpenMOSS-Team/MOSS-Transcribe-Diarize", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Sync model card with GitHub README
Browse filesSync the model card with the GitHub README (https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/main/README.md), adapted for the Hugging Face context:
- Serving order now matches GitHub: SGLang Omni (recommended) is presented before vLLM, each in its own section, including the transcription API parameter table and single-H100 benchmark results.
- Logos are referenced via absolute raw.githubusercontent.com URLs; the architecture image uses the Model_Architecture.png hosted in this repo.
- Relative GitHub links (e.g. examples/prompts.md) converted to absolute URLs.
- Kept the HF YAML metadata, the trust_remote_code note, and the MLC-SLM challenge link from the current card.
- Dropped the HF self-badge and the Trendshift badge; added the GitHub badge.
|
@@ -20,62 +20,85 @@ pipeline_tag: audio-text-to-text
|
|
| 20 |
|
| 21 |
# MOSS-Transcribe-Diarize
|
| 22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
<div align="center">
|
| 24 |
<a href="https://github.com/OpenMOSS/MOSS-Transcribe-Diarize"><img src="https://img.shields.io/badge/GitHub-OpenMOSS%2FMOSS--Transcribe--Diarize-black?logo=github"></a>
|
| 25 |
-
<a href="https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize"><img src="https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface"></a>
|
| 26 |
<a href="https://arxiv.org/abs/2601.01554"><img src="https://img.shields.io/badge/arXiv-2601.01554-b31b1b?logo=arxiv"></a>
|
|
|
|
| 27 |
</div>
|
| 28 |
|
| 29 |
-
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness.
|
| 30 |
-
|
| 31 |
-
It supports transcription and diarization across 50+ languages, single-pass inference on audio recordings up to 90 minutes long, and custom hotword prompting for domain-specific terms.
|
| 32 |
-
|
| 33 |
-
Given an audio or video file, the model generates a compact speaker-aware transcript in one pass, including timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and beyond.
|
| 34 |
|
| 35 |
## News
|
| 36 |
|
| 37 |
-
* 2026-07-14: 🏆 MOSS-Transcribe-Diarize won first place in the [2nd MLC-SLM Challenge](https://www.nexdata.ai/competition/mlc-slm) at INTERSPEECH 2026,
|
| 38 |
-
* 2026-07-09:
|
| 39 |
|
| 40 |
## Contents
|
| 41 |
|
| 42 |
- [Introduction](#introduction)
|
| 43 |
- [Model Architecture](#model-architecture)
|
| 44 |
- [Evaluation](#evaluation)
|
|
|
|
| 45 |
- [Quickstart](#quickstart)
|
| 46 |
- [Environment Setup](#environment-setup)
|
| 47 |
- [Python Usage](#python-usage)
|
| 48 |
-
- [Serve with
|
|
|
|
|
|
|
| 49 |
- [Subtitle Web App](#subtitle-web-app)
|
| 50 |
-
- [Output Format](#output-format)
|
| 51 |
-
- [More Information](#more-information)
|
| 52 |
-
- [License](#license)
|
| 53 |
- [Citation](#citation)
|
|
|
|
| 54 |
|
| 55 |
## Introduction
|
| 56 |
|
| 57 |
-
MOSS-Transcribe-Diarize
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
-
The model
|
| 60 |
|
| 61 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
|
| 67 |
## Model Architecture
|
| 68 |
|
| 69 |
<p align="center">
|
| 70 |
-
<img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize
|
| 71 |
</p>
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`.
|
| 75 |
|
| 76 |
## Evaluation
|
| 77 |
|
| 78 |
-
|
|
|
|
|
|
|
| 79 |
|
| 80 |
<div style="overflow-x: auto;">
|
| 81 |
<table style="white-space: nowrap;">
|
|
@@ -159,17 +182,14 @@ We evaluate MOSS-Transcribe-Diarize using three objective metrics: Character Err
|
|
| 159 |
|
| 160 |
### Environment Setup
|
| 161 |
|
| 162 |
-
Use a clean Python environment. The
|
| 163 |
|
| 164 |
```bash
|
| 165 |
-
conda create -n moss-transcribe-diarize python=3.12 -y
|
| 166 |
-
conda activate moss-transcribe-diarize
|
| 167 |
-
|
| 168 |
git clone https://github.com/OpenMOSS/MOSS-Transcribe-Diarize.git
|
| 169 |
cd MOSS-Transcribe-Diarize
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
pip install -e .
|
| 173 |
```
|
| 174 |
|
| 175 |
The GitHub package provides helper utilities such as audio/video loading, transcription message construction, transcript parsing, CLI inference, and the subtitle web app. The model weights and remote-code model files are loaded from this Hugging Face repository.
|
|
@@ -198,11 +218,7 @@ model = AutoModelForCausalLM.from_pretrained(
|
|
| 198 |
trust_remote_code=True,
|
| 199 |
dtype="auto",
|
| 200 |
).to(dtype=dtype).to(device).eval()
|
| 201 |
-
|
| 202 |
-
processor = AutoProcessor.from_pretrained(
|
| 203 |
-
model_id,
|
| 204 |
-
trust_remote_code=True,
|
| 205 |
-
)
|
| 206 |
|
| 207 |
messages = build_transcription_messages(audio_path)
|
| 208 |
result = generate_transcription(
|
|
@@ -221,30 +237,109 @@ for segment in parse_transcript(result["text"]):
|
|
| 221 |
print(segment.start, segment.end, segment.speaker, segment.text)
|
| 222 |
```
|
| 223 |
|
| 224 |
-
The message flow follows the common Qwen multimodal pattern:
|
| 225 |
|
| 226 |
1. `processor.apply_chat_template(messages, tokenize=False)` renders text with audio placeholders.
|
| 227 |
-
2.
|
| 228 |
3. `processor(text=text, audio=audios)` computes Whisper input features and expands audio placeholders.
|
| 229 |
4. `model.generate(...)` produces timestamped transcription and diarization text.
|
| 230 |
|
| 231 |
-
###
|
| 232 |
|
| 233 |
-
|
| 234 |
|
| 235 |
-
|
| 236 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 237 |
```
|
| 238 |
|
| 239 |
-
|
| 240 |
|
| 241 |
-
```
|
| 242 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
```
|
| 244 |
|
| 245 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
|
| 247 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
MOSS-Transcribe-Diarize supports vLLM serving through the OpenAI-compatible transcription API. Use a pinned vLLM nightly build that includes the MOSS-Transcribe-Diarize model registration. Choose one of the following commands: for CUDA 12 environments, use `cu129`; for CUDA 13 environments, use `cu130`.
|
| 250 |
|
|
@@ -274,47 +369,25 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
|
| 274 |
-F temperature="0"
|
| 275 |
```
|
| 276 |
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
```bash
|
| 280 |
-
hf download OpenMOSS-Team/MOSS-Transcribe-Diarize
|
| 281 |
-
```
|
| 282 |
-
|
| 283 |
-
Serve the model:
|
| 284 |
-
|
| 285 |
-
```bash
|
| 286 |
-
sgl-omni serve \
|
| 287 |
-
--model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 288 |
-
--port 8000 \
|
| 289 |
-
--max-running-requests 16 \
|
| 290 |
-
--cuda-graph-max-bs 16 \
|
| 291 |
-
--mem-fraction-static 0.80
|
| 292 |
-
```
|
| 293 |
|
| 294 |
-
|
| 295 |
|
| 296 |
-
```
|
| 297 |
-
|
| 298 |
-
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 299 |
-
-F file=@audio.wav \
|
| 300 |
-
-F response_format=verbose_json
|
| 301 |
```
|
| 302 |
|
| 303 |
-
|
| 304 |
|
| 305 |
-
```
|
| 306 |
-
|
| 307 |
-
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 308 |
-
-F file=@audio.wav \
|
| 309 |
-
-F response_format=verbose_json \
|
| 310 |
-
-F max_new_tokens=65536
|
| 311 |
```
|
| 312 |
|
| 313 |
-
|
| 314 |
|
| 315 |
### Subtitle Web App
|
| 316 |
|
| 317 |
-
The
|
| 318 |
|
| 319 |
```bash
|
| 320 |
mtd-subtitle-web \
|
|
@@ -334,39 +407,9 @@ mtd-subtitle /path/to/input.mp4 \
|
|
| 334 |
--render
|
| 335 |
```
|
| 336 |
|
| 337 |
-
## Output Format
|
| 338 |
-
|
| 339 |
-
The canonical output format is:
|
| 340 |
-
|
| 341 |
-
```text
|
| 342 |
-
[start_time][Sxx]transcribed speech[end_time]
|
| 343 |
-
```
|
| 344 |
-
|
| 345 |
-
Example:
|
| 346 |
-
|
| 347 |
-
```text
|
| 348 |
-
[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
|
| 349 |
-
```
|
| 350 |
-
|
| 351 |
-
In this format:
|
| 352 |
-
|
| 353 |
-
* `start_time` and `end_time` are timestamps in seconds.
|
| 354 |
-
* `[S01]`, `[S02]`, and similar labels are anonymous model-generated speaker labels.
|
| 355 |
-
* Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.
|
| 356 |
-
|
| 357 |
-
## More Information
|
| 358 |
-
|
| 359 |
-
* **GitHub**: <https://github.com/OpenMOSS/MOSS-Transcribe-Diarize>
|
| 360 |
-
* **MOSI.AI**: <https://mosi.cn>
|
| 361 |
-
* **OpenMOSS**: <https://www.open-moss.com>
|
| 362 |
-
|
| 363 |
-
## License
|
| 364 |
-
|
| 365 |
-
MOSS-Transcribe-Diarize 0.9B is licensed under the Apache License 2.0.
|
| 366 |
-
|
| 367 |
## Citation
|
| 368 |
|
| 369 |
-
If you use MOSS-Transcribe-Diarize
|
| 370 |
|
| 371 |
```bibtex
|
| 372 |
@misc{moss_transcribe_diarize_2026,
|
|
@@ -379,3 +422,11 @@ If you use MOSS-Transcribe-Diarize 0.9B, please cite the technical report:
|
|
| 379 |
url={https://arxiv.org/abs/2601.01554}
|
| 380 |
}
|
| 381 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
# MOSS-Transcribe-Diarize
|
| 22 |
|
| 23 |
+
<br>
|
| 24 |
+
|
| 25 |
+
<p align="center">
|
| 26 |
+
<img src="https://raw.githubusercontent.com/OpenMOSS/MOSS-Transcribe-Diarize/main/assets/OpenMOSS_Logo.png" height="70" align="middle" />
|
| 27 |
+
|
| 28 |
+
<img src="https://raw.githubusercontent.com/OpenMOSS/MOSS-Transcribe-Diarize/main/assets/mosi-logo.png" height="50" align="middle" />
|
| 29 |
+
</p>
|
| 30 |
+
|
| 31 |
<div align="center">
|
| 32 |
<a href="https://github.com/OpenMOSS/MOSS-Transcribe-Diarize"><img src="https://img.shields.io/badge/GitHub-OpenMOSS%2FMOSS--Transcribe--Diarize-black?logo=github"></a>
|
|
|
|
| 33 |
<a href="https://arxiv.org/abs/2601.01554"><img src="https://img.shields.io/badge/arXiv-2601.01554-b31b1b?logo=arxiv"></a>
|
| 34 |
+
<a href="https://x.com/MosiAI_Official"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&"></a>
|
| 35 |
</div>
|
| 36 |
|
| 37 |
+
MOSS-Transcribe-Diarize 0.9B is an open-source SOTA end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness. MOSS-Transcribe-Diarize Pro is a stronger model with higher overall performance and will be available through API access soon.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
## News
|
| 40 |
|
| 41 |
+
* 2026-07-14: 🏆 MOSS-Transcribe-Diarize won first place in the [2nd MLC-SLM Challenge](https://www.nexdata.ai/competition/mlc-slm) at INTERSPEECH 2026, covering 14 languages.
|
| 42 |
+
* 2026-07-09: Open-sourced MOSS-Transcribe-Diarize 0.9B.
|
| 43 |
|
| 44 |
## Contents
|
| 45 |
|
| 46 |
- [Introduction](#introduction)
|
| 47 |
- [Model Architecture](#model-architecture)
|
| 48 |
- [Evaluation](#evaluation)
|
| 49 |
+
- [Objective Evaluation](#objective-evaluation)
|
| 50 |
- [Quickstart](#quickstart)
|
| 51 |
- [Environment Setup](#environment-setup)
|
| 52 |
- [Python Usage](#python-usage)
|
| 53 |
+
- [Serve with SGLang Omni](#serve-with-sglang-omni)
|
| 54 |
+
- [Serve with vLLM](#serve-with-vllm)
|
| 55 |
+
- [Custom Prompt and Hotwords](#custom-prompt-and-hotwords)
|
| 56 |
- [Subtitle Web App](#subtitle-web-app)
|
|
|
|
|
|
|
|
|
|
| 57 |
- [Citation](#citation)
|
| 58 |
+
- [Star History](#star-history)
|
| 59 |
|
| 60 |
## Introduction
|
| 61 |
|
| 62 |
+
MOSS-Transcribe-Diarize is our flagship SOTA model family for turning real-world long-form audio into structured, speaker-aware transcripts in one pass. Instead of stitching together separate ASR and diarization systems, these models jointly perform speech transcription and speaker diarization, producing time-aligned text with precise timestamps and consistent speaker labels such as `[S01]`, `[S02]`, and beyond.
|
| 63 |
+
|
| 64 |
+
Built for meetings, calls, podcasts, interviews, lectures, and video content, MOSS-Transcribe-Diarize is designed to handle long, messy, multi-speaker recordings where reliability matters. It can also emit optional acoustic event annotations, giving downstream systems a richer understanding of what happened, who spoke, and when.
|
| 65 |
+
|
| 66 |
+
MOSS-Transcribe-Diarize supports 50+ languages.
|
| 67 |
|
| 68 |
+
The model accepts raw audio and emits a compact timestamped transcript. The canonical output format is:
|
| 69 |
|
| 70 |
+
```text
|
| 71 |
+
[start_time][Sxx]transcribed speech[end_time]
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
Timestamps are expressed in seconds, and adjacent segments are concatenated into a single stream, for example:
|
| 75 |
|
| 76 |
+
```text
|
| 77 |
+
[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
|
| 78 |
+
```
|
| 79 |
|
| 80 |
## Model Architecture
|
| 81 |
|
| 82 |
<p align="center">
|
| 83 |
+
<img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize model architecture" width="90%" />
|
| 84 |
</p>
|
| 85 |
|
| 86 |
+
| Component | Specification |
|
| 87 |
+
|---|---|
|
| 88 |
+
| Text backbone | Qwen3-0.6B style causal decoder |
|
| 89 |
+
| Audio encoder | Whisper-Medium encoder configuration |
|
| 90 |
+
| Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
|
| 91 |
+
| Audio-text bridge | 4x temporal merge + MLP adaptor |
|
| 92 |
+
| Fusion | Audio features replace <code><|audio_pad|></code> embeddings via `masked_scatter` |
|
| 93 |
+
| Output format | Compact `[start][Sxx]text[end]` transcript with speaker tags such as `[S01]` |
|
| 94 |
|
| 95 |
This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`.
|
| 96 |
|
| 97 |
## Evaluation
|
| 98 |
|
| 99 |
+
### Objective Evaluation
|
| 100 |
+
|
| 101 |
+
We evaluate MOSS-Transcribe-Diarize using three objective metrics: Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Δcp. Lower is better for all metrics. Best results are bolded, second-best results are underlined. A dash (`-`) indicates that the result is unavailable.
|
| 102 |
|
| 103 |
<div style="overflow-x: auto;">
|
| 104 |
<table style="white-space: nowrap;">
|
|
|
|
| 182 |
|
| 183 |
### Environment Setup
|
| 184 |
|
| 185 |
+
Use a clean Python environment. The project is tested with Python 3.12 and Transformers 5.x.
|
| 186 |
|
| 187 |
```bash
|
|
|
|
|
|
|
|
|
|
| 188 |
git clone https://github.com/OpenMOSS/MOSS-Transcribe-Diarize.git
|
| 189 |
cd MOSS-Transcribe-Diarize
|
| 190 |
+
uv venv --python 3.12 .venv
|
| 191 |
+
source .venv/bin/activate
|
| 192 |
+
uv pip install -e ".[torch-runtime]" --torch-backend=auto
|
| 193 |
```
|
| 194 |
|
| 195 |
The GitHub package provides helper utilities such as audio/video loading, transcription message construction, transcript parsing, CLI inference, and the subtitle web app. The model weights and remote-code model files are loaded from this Hugging Face repository.
|
|
|
|
| 218 |
trust_remote_code=True,
|
| 219 |
dtype="auto",
|
| 220 |
).to(dtype=dtype).to(device).eval()
|
| 221 |
+
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
|
| 223 |
messages = build_transcription_messages(audio_path)
|
| 224 |
result = generate_transcription(
|
|
|
|
| 237 |
print(segment.start, segment.end, segment.speaker, segment.text)
|
| 238 |
```
|
| 239 |
|
| 240 |
+
The message flow follows the common Qwen multimodal pattern. The chat template is loaded from the model by `AutoProcessor`:
|
| 241 |
|
| 242 |
1. `processor.apply_chat_template(messages, tokenize=False)` renders text with audio placeholders.
|
| 243 |
+
2. `process_audio_info(messages, sampling_rate)` loads audio waveforms from the same messages.
|
| 244 |
3. `processor(text=text, audio=audios)` computes Whisper input features and expands audio placeholders.
|
| 245 |
4. `model.generate(...)` produces timestamped transcription and diarization text.
|
| 246 |
|
| 247 |
+
### Serve with SGLang Omni
|
| 248 |
|
| 249 |
+
[SGLang Omni](https://github.com/sgl-project/sglang-omni) is the recommended serving backend for MOSS-Transcribe-Diarize, providing optimized long-form audio inference through the OpenAI-compatible `/v1/audio/transcriptions` endpoint.
|
| 250 |
|
| 251 |
+
SGLang Omni currently targets CUDA 13 environments. Please follow the official [installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md) for the supported setup. For CUDA 12 environments, the vLLM workflow is also available below.
|
| 252 |
+
|
| 253 |
+
Download the model:
|
| 254 |
+
|
| 255 |
+
```bash
|
| 256 |
+
hf download OpenMOSS-Team/MOSS-Transcribe-Diarize
|
| 257 |
```
|
| 258 |
|
| 259 |
+
Serve the model:
|
| 260 |
|
| 261 |
+
```bash
|
| 262 |
+
sgl-omni serve \
|
| 263 |
+
--model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 264 |
+
--port 8000 \
|
| 265 |
+
--max-running-requests 16 \
|
| 266 |
+
--cuda-graph-max-bs 16 \
|
| 267 |
+
--mem-fraction-static 0.80
|
| 268 |
+
```
|
| 269 |
+
|
| 270 |
+
Use `response_format=verbose_json` when you need parsed speaker segments. `json` returns the raw transcript text only.
|
| 271 |
+
|
| 272 |
+
```bash
|
| 273 |
+
curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
| 274 |
+
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 275 |
+
-F file=@audio.wav \
|
| 276 |
+
-F response_format=verbose_json
|
| 277 |
+
```
|
| 278 |
+
|
| 279 |
+
```python
|
| 280 |
+
import requests
|
| 281 |
+
|
| 282 |
+
with open("audio.wav", "rb") as f:
|
| 283 |
+
resp = requests.post(
|
| 284 |
+
"http://localhost:8000/v1/audio/transcriptions",
|
| 285 |
+
data={
|
| 286 |
+
"model": "OpenMOSS-Team/MOSS-Transcribe-Diarize",
|
| 287 |
+
"response_format": "verbose_json",
|
| 288 |
+
},
|
| 289 |
+
files={"file": ("audio.wav", f, "audio/wav")},
|
| 290 |
+
timeout=300,
|
| 291 |
+
)
|
| 292 |
+
|
| 293 |
+
resp.raise_for_status()
|
| 294 |
+
payload = resp.json()
|
| 295 |
+
print(payload["text"])
|
| 296 |
+
for segment in payload.get("segments", []):
|
| 297 |
+
print(f"[{segment['start']:.2f}-{segment['end']:.2f}] {segment['text']}")
|
| 298 |
+
```
|
| 299 |
+
|
| 300 |
+
For longer multi-speaker audio, raise `max_new_tokens` so the decoder can finish the full diarized transcript:
|
| 301 |
+
|
| 302 |
+
```bash
|
| 303 |
+
curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
| 304 |
+
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 305 |
+
-F file=@audio.wav \
|
| 306 |
+
-F response_format=verbose_json \
|
| 307 |
+
-F max_new_tokens=65536
|
| 308 |
```
|
| 309 |
|
| 310 |
+
| Parameter | Type | Default | Description |
|
| 311 |
+
|---|---|---|---|
|
| 312 |
+
| `file` | file | required | Audio file uploaded as multipart form data |
|
| 313 |
+
| `model` | string | server default | Model identifier |
|
| 314 |
+
| `language` | string | unset | Optional language hint |
|
| 315 |
+
| `response_format` | string | `json` | `json`, `verbose_json`, or `text` |
|
| 316 |
+
| `temperature` | float | model default (`0.0`) | Sampling temperature |
|
| 317 |
+
| `max_new_tokens` | int | `5120` | Max generated tokens; raise for long audio, for example `65536` |
|
| 318 |
+
| `prompt` | string | unset | Optional instruction override; omit to use the built-in transcribe+diarize prompt |
|
| 319 |
|
| 320 |
+
For benchmarking, performance numbers, and implementation details, see the [SGLang Omni cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/moss_transcribe_diarize.md). The following single-H100 results are reported for short- and long-sequence multi-speaker ASR tasks.
|
| 321 |
+
|
| 322 |
+
`movies` short-sequence ASR:
|
| 323 |
+
|
| 324 |
+
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 325 |
+
|---:|---:|---:|---:|---:|
|
| 326 |
+
| 1 | 2.57 | 0.388 | 0.0612 | 29.76 |
|
| 327 |
+
| 2 | 4.89 | 0.409 | 0.0659 | 56.55 |
|
| 328 |
+
| 4 | 6.62 | 0.513 | 0.0790 | 76.64 |
|
| 329 |
+
| 8 | 6.80 | 0.533 | 0.0810 | 78.70 |
|
| 330 |
+
| 16 | 7.08 | 0.659 | 0.0922 | 81.98 |
|
| 331 |
+
|
| 332 |
+
`aishell4_long` long-sequence ASR:
|
| 333 |
+
|
| 334 |
+
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 335 |
+
|---:|---:|---:|---:|---:|
|
| 336 |
+
| 1 | 0.022 | 45.2 | 0.0197 | 50.64 |
|
| 337 |
+
| 2 | 0.032 | 60.7 | 0.0265 | 74.25 |
|
| 338 |
+
| 4 | 0.036 | 105.6 | 0.0461 | 81.64 |
|
| 339 |
+
| 8 | 0.040 | 172.6 | 0.0754 | 90.62 |
|
| 340 |
+
| 16 | 0.043 | 282.8 | 0.1237 | 98.83 |
|
| 341 |
+
|
| 342 |
+
### Serve with vLLM
|
| 343 |
|
| 344 |
MOSS-Transcribe-Diarize supports vLLM serving through the OpenAI-compatible transcription API. Use a pinned vLLM nightly build that includes the MOSS-Transcribe-Diarize model registration. Choose one of the following commands: for CUDA 12 environments, use `cu129`; for CUDA 13 environments, use `cu130`.
|
| 345 |
|
|
|
|
| 369 |
-F temperature="0"
|
| 370 |
```
|
| 371 |
|
| 372 |
+
### Custom Prompt and Hotwords
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 373 |
|
| 374 |
+
The default prompt is optimized for timestamped transcription and speaker diarization:
|
| 375 |
|
| 376 |
+
```text
|
| 377 |
+
请将音频转写为文本,每一段需以起始时间戳和说话人编号([S01]、[S02]、[S03]…)开头,正文为对应的语音内容,并在段末标注结束时间戳,以清晰标明该段语音范围。
|
|
|
|
|
|
|
|
|
|
| 378 |
```
|
| 379 |
|
| 380 |
+
To add hotwords, append a short hint to the default prompt:
|
| 381 |
|
| 382 |
+
```text
|
| 383 |
+
请将音频转写为文本,每一段需以起始时间戳和说话人编号([S01]、[S02]、[S03]…)开头,正文为对应的语音内容,并在段末标注结束时间戳,以清晰标明该段语音范围。热词提示:热词1, 热词2, 热词3
|
|
|
|
|
|
|
|
|
|
|
|
|
| 384 |
```
|
| 385 |
|
| 386 |
+
More prompt recipes are available in [examples/prompts.md](https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/main/examples/prompts.md). The same prompt can be passed to `build_transcription_messages`, `mtd-subtitle`, and `mtd-subtitle-web`.
|
| 387 |
|
| 388 |
### Subtitle Web App
|
| 389 |
|
| 390 |
+
The package also includes a local subtitle workflow for upload, review, subtitle export, and optional FFmpeg burn-in:
|
| 391 |
|
| 392 |
```bash
|
| 393 |
mtd-subtitle-web \
|
|
|
|
| 407 |
--render
|
| 408 |
```
|
| 409 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 410 |
## Citation
|
| 411 |
|
| 412 |
+
If you use MOSS-Transcribe-Diarize, please cite the technical report:
|
| 413 |
|
| 414 |
```bibtex
|
| 415 |
@misc{moss_transcribe_diarize_2026,
|
|
|
|
| 422 |
url={https://arxiv.org/abs/2601.01554}
|
| 423 |
}
|
| 424 |
```
|
| 425 |
+
|
| 426 |
+
## Star History
|
| 427 |
+
|
| 428 |
+
<p align="center">
|
| 429 |
+
<a href="https://www.star-history.com/#OpenMOSS/MOSS-Transcribe-Diarize&Date">
|
| 430 |
+
<img width="700" alt="Star History Chart" src="https://api.star-history.com/svg?repos=OpenMOSS%2FMOSS-Transcribe-Diarize&type=Date">
|
| 431 |
+
</a>
|
| 432 |
+
</p>
|