Audio-Text-to-Text
Transformers
Safetensors
English
Chinese
moss_transcribe_diarize
text-generation
moss
audio
speech
asr
diarization
timestamp-asr
long-form-audio
multimodal
multilingual
custom_code
Eval Results
Instructions to use OpenMOSS-Team/MOSS-Transcribe-Diarize with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMOSS-Team/MOSS-Transcribe-Diarize with Transformers:
# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("OpenMOSS-Team/MOSS-Transcribe-Diarize", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Revert README to 7f2c1bd
Browse files
README.md
CHANGED
|
@@ -39,11 +39,10 @@ Given an audio or video file, the model generates a compact speaker-aware transc
|
|
| 39 |
- [Model Architecture](#model-architecture)
|
| 40 |
- [Evaluation](#evaluation)
|
| 41 |
- [Quickstart](#quickstart)
|
| 42 |
-
- [
|
| 43 |
-
- [Serving with Hugging Face](#serving-with-native-hugging-face-transformers)
|
| 44 |
- [Python Usage](#python-usage)
|
| 45 |
- [Custom Prompt and Hotwords](#custom-prompt-and-hotwords)
|
| 46 |
-
- [Serve with vLLM](#serve-with-vllm)
|
| 47 |
- [Subtitle Web App](#subtitle-web-app)
|
| 48 |
- [Output Format](#output-format)
|
| 49 |
- [More Information](#more-information)
|
|
@@ -163,102 +162,7 @@ We evaluate MOSS-Transcribe-Diarize using three objective metrics: Character Err
|
|
| 163 |
|
| 164 |
## Quickstart
|
| 165 |
|
| 166 |
-
###
|
| 167 |
-
|
| 168 |
-
The recommended way to serve MOSS-Transcribe-Diarize is [SGLang Omni](https://github.com/sgl-project/sglang-omni) through the OpenAI-compatible `/v1/audio/transcriptions` endpoint. Install `sglang-omni` by following the [Installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md), then download the model:
|
| 169 |
-
|
| 170 |
-
```bash
|
| 171 |
-
hf download OpenMOSS-Team/MOSS-Transcribe-Diarize
|
| 172 |
-
```
|
| 173 |
-
|
| 174 |
-
Serve the model:
|
| 175 |
-
|
| 176 |
-
```bash
|
| 177 |
-
sgl-omni serve \
|
| 178 |
-
--model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 179 |
-
--port 8000 \
|
| 180 |
-
--max-running-requests 16 \
|
| 181 |
-
--cuda-graph-max-bs 16 \
|
| 182 |
-
--mem-fraction-static 0.80
|
| 183 |
-
```
|
| 184 |
-
|
| 185 |
-
Use `response_format=verbose_json` when you need parsed speaker segments. `json` returns the raw transcript text only.
|
| 186 |
-
|
| 187 |
-
```bash
|
| 188 |
-
curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
| 189 |
-
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 190 |
-
-F file=@audio.wav \
|
| 191 |
-
-F response_format=verbose_json
|
| 192 |
-
```
|
| 193 |
-
|
| 194 |
-
```python
|
| 195 |
-
import requests
|
| 196 |
-
|
| 197 |
-
with open("audio.wav", "rb") as f:
|
| 198 |
-
resp = requests.post(
|
| 199 |
-
"http://localhost:8000/v1/audio/transcriptions",
|
| 200 |
-
data={
|
| 201 |
-
"model": "OpenMOSS-Team/MOSS-Transcribe-Diarize",
|
| 202 |
-
"response_format": "verbose_json",
|
| 203 |
-
},
|
| 204 |
-
files={"file": ("audio.wav", f, "audio/wav")},
|
| 205 |
-
timeout=300,
|
| 206 |
-
)
|
| 207 |
-
|
| 208 |
-
resp.raise_for_status()
|
| 209 |
-
payload = resp.json()
|
| 210 |
-
print(payload["text"])
|
| 211 |
-
for segment in payload.get("segments", []):
|
| 212 |
-
print(
|
| 213 |
-
f"[{segment['start']:.2f}-{segment['end']:.2f}] {segment['text']}"
|
| 214 |
-
)
|
| 215 |
-
```
|
| 216 |
-
|
| 217 |
-
For longer multi-speaker audio, raise `max_new_tokens` so the decoder can finish the full diarized transcript:
|
| 218 |
-
|
| 219 |
-
```bash
|
| 220 |
-
curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
| 221 |
-
-F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 222 |
-
-F file=@audio.wav \
|
| 223 |
-
-F response_format=verbose_json \
|
| 224 |
-
-F max_new_tokens=65536
|
| 225 |
-
```
|
| 226 |
-
|
| 227 |
-
| Parameter | Type | Default | Description |
|
| 228 |
-
|---|---|---|---|
|
| 229 |
-
| `file` | file | required | Audio file uploaded as multipart form data |
|
| 230 |
-
| `model` | string | server default | Model identifier |
|
| 231 |
-
| `language` | string | unset | Optional language hint |
|
| 232 |
-
| `response_format` | string | `json` | `json`, `verbose_json`, or `text` |
|
| 233 |
-
| `temperature` | float | model default (`0.0`) | Sampling temperature |
|
| 234 |
-
| `max_new_tokens` | int | `5120` | Max generated tokens; raise for long audio (e.g. `65536`) |
|
| 235 |
-
| `prompt` | string | unset | Optional instruction override; omit to use the built-in transcribe+diarize prompt |
|
| 236 |
-
|
| 237 |
-
For benchmarking, performance numbers, and more details, see the [SGLang Omni cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/moss_transcribe_diarize.md), here we list the performance of short/long-sequence mutli-speaker ASR tasks on single H100:
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
1. `movies800times` for short sequence ASR:
|
| 241 |
-
|
| 242 |
-
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 243 |
-
|---:|---:|---:|---:|---:|
|
| 244 |
-
| 1 | 2.57 | 0.388 | 0.0612 | 29.76 |
|
| 245 |
-
| 2 | 4.89 | 0.409 | 0.0659 | 56.55 |
|
| 246 |
-
| 4 | 6.62 | 0.513 | 0.0790 | 76.64 |
|
| 247 |
-
| 8 | 6.80 | 0.533 | 0.0810 | 78.70 |
|
| 248 |
-
| 16 | 7.08 | 0.659 | 0.0922 | 81.98 |
|
| 249 |
-
|
| 250 |
-
2. `aishell4_long` for long sequence ASR:
|
| 251 |
-
|
| 252 |
-
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 253 |
-
|---:|---:|---:|---:|---:|
|
| 254 |
-
| 1 | 0.022 | 45.2 | 0.0197 | 50.64 |
|
| 255 |
-
| 2 | 0.032 | 60.7 | 0.0265 | 74.25 |
|
| 256 |
-
| 4 | 0.036 | 105.6 | 0.0461 | 81.64 |
|
| 257 |
-
| 8 | 0.040 | 172.6 | 0.0754 | 90.62 |
|
| 258 |
-
| 16 | 0.043 | 282.8 | 0.1237 | 98.83 |
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
### Serving with Native Hugging Face Transformers
|
| 262 |
|
| 263 |
Use a clean Python environment. The model uses custom Transformers code, so load the model and processor with `trust_remote_code=True`.
|
| 264 |
|
|
@@ -345,10 +249,9 @@ To add hotwords, append a short hint to the default prompt:
|
|
| 345 |
|
| 346 |
More prompt recipes are available in the GitHub repository: <https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/main/examples/prompts.md>
|
| 347 |
|
|
|
|
| 348 |
|
| 349 |
-
|
| 350 |
-
|
| 351 |
-
MOSS-Transcribe-Diarize also supports vLLM serving through the OpenAI-compatible transcription API. Use a pinned vLLM nightly build that includes the MOSS-Transcribe-Diarize model registration. Choose one of the following commands: for CUDA 12 environments, use `cu129`; for CUDA 13 environments, use `cu130`.
|
| 352 |
|
| 353 |
```bash
|
| 354 |
uv pip install -U vllm \
|
|
@@ -376,6 +279,25 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
|
| 376 |
-F temperature="0"
|
| 377 |
```
|
| 378 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 379 |
### Subtitle Web App
|
| 380 |
|
| 381 |
The source package includes a local subtitle workflow for upload, review, subtitle export, and optional FFmpeg burn-in:
|
|
|
|
| 39 |
- [Model Architecture](#model-architecture)
|
| 40 |
- [Evaluation](#evaluation)
|
| 41 |
- [Quickstart](#quickstart)
|
| 42 |
+
- [Environment Setup](#environment-setup)
|
|
|
|
| 43 |
- [Python Usage](#python-usage)
|
| 44 |
- [Custom Prompt and Hotwords](#custom-prompt-and-hotwords)
|
| 45 |
+
- [Serve with vLLM and SGLang](#serve-with-vllm-and-sglang)
|
| 46 |
- [Subtitle Web App](#subtitle-web-app)
|
| 47 |
- [Output Format](#output-format)
|
| 48 |
- [More Information](#more-information)
|
|
|
|
| 162 |
|
| 163 |
## Quickstart
|
| 164 |
|
| 165 |
+
### Environment Setup
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
|
| 167 |
Use a clean Python environment. The model uses custom Transformers code, so load the model and processor with `trust_remote_code=True`.
|
| 168 |
|
|
|
|
| 249 |
|
| 250 |
More prompt recipes are available in the GitHub repository: <https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/main/examples/prompts.md>
|
| 251 |
|
| 252 |
+
### Serve with vLLM and SGLang
|
| 253 |
|
| 254 |
+
MOSS-Transcribe-Diarize supports vLLM serving through the OpenAI-compatible transcription API. Use a pinned vLLM nightly build that includes the MOSS-Transcribe-Diarize model registration. Choose one of the following commands: for CUDA 12 environments, use `cu129`; for CUDA 13 environments, use `cu130`.
|
|
|
|
|
|
|
| 255 |
|
| 256 |
```bash
|
| 257 |
uv pip install -U vllm \
|
|
|
|
| 279 |
-F temperature="0"
|
| 280 |
```
|
| 281 |
|
| 282 |
+
The same request format can be used with an SGLang OpenAI-compatible server:
|
| 283 |
+
|
| 284 |
+
```bash
|
| 285 |
+
python -m sglang.launch_server \
|
| 286 |
+
--model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 287 |
+
--served-model-name OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
| 288 |
+
--trust-remote-code \
|
| 289 |
+
--host 0.0.0.0 \
|
| 290 |
+
--port 30000
|
| 291 |
+
```
|
| 292 |
+
|
| 293 |
+
```bash
|
| 294 |
+
curl http://localhost:30000/v1/audio/transcriptions \
|
| 295 |
+
-F model="OpenMOSS-Team/MOSS-Transcribe-Diarize" \
|
| 296 |
+
-F file=@"audio.wav" \
|
| 297 |
+
-F response_format="json" \
|
| 298 |
+
-F temperature="0"
|
| 299 |
+
```
|
| 300 |
+
|
| 301 |
### Subtitle Web App
|
| 302 |
|
| 303 |
The source package includes a local subtitle workflow for upload, review, subtitle export, and optional FFmpeg burn-in:
|