Update model card README

README.md

@@ -1,5 +1,6 @@
 ---
 license: apache-2.0
+library_name: transformers
 language:
 - en
 - zh
@@ -12,46 +13,56 @@ tags:
 - timestamp-asr
 - long-form-audio
 - multimodal
+- custom_code
 pipeline_tag: audio-text-to-text
 ---
 
 # MOSS-Transcribe-Diarize
 
-MOSS-Transcribe-Diarize is an open-source **speech transcription and diarization model** from the OpenMOSS team. It performs unified modeling over long-form, multi-speaker audio, supporting **automatic speech recognition, speaker-aware transcription, speaker diarization, timestamp prediction, and compact transcript generation**.
+<div align="center">
+  <a href="https://github.com/OpenMOSS/MOSS-Transcribe-Diarize"><img src="https://img.shields.io/badge/GitHub-OpenMOSS%2FMOSS--Transcribe--Diarize-black?logo=github"></a>
+  <a href="https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize"><img src="https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface"></a>
+  <a href="https://arxiv.org/abs/2601.01554"><img src="https://img.shields.io/badge/arXiv-2601.01554-b31b1b?logo=arxiv"></a>
+</div>
 
-Given an audio or video file, MOSS-Transcribe-Diarize generates a speaker-aware transcript in one pass, with segment timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and `[S03]`.
+MOSS-Transcribe-Diarize is an end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness.
+
+Given an audio or video file, the model generates a compact speaker-aware transcript in one pass, including timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and beyond.
 
 ## News
 
-* 2026.05.21: We have released MOSS-Transcribe-Diarize.
+* 2026-07-09: Released MOSS-Transcribe-Diarize 0.9B.
 
 ## Contents
 
 - [Introduction](#introduction)
 - [Model Architecture](#model-architecture)
-- [Released Models](#released-models)
 - [Evaluation](#evaluation)
 - [Quickstart](#quickstart)
   - [Environment Setup](#environment-setup)
-  - [Command Line Inference](#command-line-inference)
   - [Python Usage](#python-usage)
+  - [Custom Prompt and Hotwords](#custom-prompt-and-hotwords)
+  - [Serve with vLLM and SGLang](#serve-with-vllm-and-sglang)
+  - [Subtitle Web App](#subtitle-web-app)
 - [Output Format](#output-format)
 - [More Information](#more-information)
-- [LICENSE](#license)
+- [License](#license)
 - [Citation](#citation)
 
 ## Introduction
 
-Long-form speech understanding requires more than plain transcription. For meetings, calls, podcasts, interviews, lectures, videos, and other real-world recordings, a useful transcript should identify **what was said**, **who said it**, and **when it was said**.
+MOSS-Transcribe-Diarize turns real-world long-form audio into structured, speaker-aware transcripts in one pass. Instead of stitching together separate ASR and diarization systems, it jointly performs speech transcription and speaker diarization, producing time-aligned text with consistent speaker labels.
+
+The model is built for meetings, calls, podcasts, interviews, lectures, videos, and other long or messy multi-speaker recordings. It can also emit acoustic event annotations, giving downstream systems a richer view of what happened, who spoke, and when.
 
-MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model.
+Core capabilities:
 
 * **Long-form ASR**: Transcribes long audio and video recordings into text.
 * **Speaker-aware transcription**: Adds anonymous speaker labels to each speech segment.
-* **Speaker diarization**: Produces "who spoke when" style output without a separate diarization pipeline.
+* **Speaker diarization**: Produces "who spoke when" output without a separate diarization pipeline.
 * **Timestamp prediction**: Generates segment-level start and end timestamps.
-* **Audio and video input**: Supports common audio files and video containers decoded through PyAV.
-* **Promptable generation**: Allows users to customize the transcription instruction.
+* **Audio and video input**: Supports common audio files and video containers through the source package utilities.
+* **Promptable generation**: Supports custom transcription instructions and hotwords.
 
 ## Model Architecture
 
@@ -59,102 +70,117 @@ MOSS-Transcribe-Diarize is built to unify these capabilities within a single gen
   <img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize model architecture" width="900">
 </p>
 
-MOSS-Transcribe-Diarize follows a modular audio-language design comprising three components: an audio encoder, a modality adapter, and a causal language model. Raw audio is converted into log-mel features, encoded by a Whisper-style audio encoder, projected into the language model embedding space through an MLP adapter, and then consumed by a Qwen3-style causal decoder for auto-regressive text generation.
-
-The model uses audio placeholder tokens in the text sequence. During the forward pass, projected audio representations replace the corresponding placeholder embeddings, allowing the language model to generate timestamped, speaker-aware transcripts conditioned on the input audio.
-
 | Component | Specification |
-| --- | --- |
+|---|---|
 | Text backbone | Qwen3-0.6B style causal decoder |
 | Audio encoder | Whisper-Medium encoder configuration |
 | Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
-| Audio-text adapter | 4x temporal merge + MLP adapter |
-| Fusion method | Audio features replace `<|audio_pad|>` embeddings |
-| Output format | Compact `[start][Sxx]text[end]` transcript |
-
-## Released Models
+| Audio-text bridge | 4x temporal merge + MLP adaptor |
+| Fusion | Audio features replace <code>&lt;&#124;audio_pad&#124;&gt;</code> embeddings via `masked_scatter` |
+| Output format | Compact `[start][Sxx]text[end]` transcript with speaker tags such as `[S01]` |
 
-| Model | Audio Encoder | LLM Backbone | Hugging Face |
-| --- | --- | --- | --- |
-| **MOSS-Transcribe-Diarize** | Whisper-style audio encoder | Qwen3-0.6B style decoder | [Hugging Face](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) |
-
-> More model variants may be released in the future. Stay tuned!
+This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`.
 
 ## Evaluation
 
-We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates unavailable results.
-
-| Dataset | Metric | Doubao | ElevenLabs | GPT-4o | Gemini 2.5 Pro | Gemini 3 Pro | VIBEVOICE ASR | MOSS Transcribe Diarize |
-| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| **AISHELL-4** | CER down | 18.18 | 19.58 | - | 42.70 | 22.75 | 21.40 | **14.19** |
-| | cpCER down | 27.86 | 37.95 | - | 53.42 | 27.43 | 24.99 | **14.98** |
-| | Delta-cp down | 9.68 | 18.36 | - | 10.72 | 4.68 | 3.59 | **0.79** |
-| **Podcast** | CER down | 7.93 | 8.50 | - | 7.38 | - | 27.94 | **4.46** |
-| | cpCER down | 10.54 | 11.34 | - | 10.23 | - | 48.30 | **6.97** |
-| | Delta-cp down | 2.61 | 2.85 | - | 2.85 | - | 20.36 | **2.50** |
-| **Movies** | CER down | 9.94 | 11.49 | 14.37 | 15.46 | 8.62 | 14.59 | **6.58** |
-| | cpCER down | 30.88 | 17.85 | 23.67 | 24.15 | 14.73 | 42.54 | **13.68** |
-| | Delta-cp down | 20.94 | **6.37** | 9.31 | 8.69 | 6.11 | 27.94 | 7.24 |
-| **Alimeeting** | CER down | 25.25 | 25.70 | - | 27.43 | 26.75 | 27.40 | **24.80** |
-| | cpCER down | 37.57 | 36.69 | - | 41.64 | 32.84 | 29.33 | **21.51** |
-| | Delta-cp down | 12.31 | 10.99 | - | 14.21 | 6.09 | 1.93 | **-0.33** |
+We evaluate MOSS-Transcribe-Diarize using three objective metrics: Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates that the result is unavailable.
+
+<div style="overflow-x: auto;">
+<table style="white-space: nowrap;">
+  <thead>
+    <tr>
+      <th rowspan="2" style="min-width: 220px;">Model</th>
+      <th colspan="3" style="text-align:center;">AISHELL&#8209;4</th>
+      <th colspan="3" style="text-align:center;">Alimeeting</th>
+      <th colspan="3" style="text-align:center;">Podcast</th>
+      <th colspan="3" style="text-align:center;">Movies</th>
+    </tr>
+    <tr>
+      <th>CER↓</th><th>cpCER↓</th><th>Δcp↓</th>
+      <th>CER↓</th><th>cpCER↓</th><th>Δcp↓</th>
+      <th>CER↓</th><th>cpCER↓</th><th>Δcp↓</th>
+      <th>CER↓</th><th>cpCER↓</th><th>Δcp↓</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style="white-space: nowrap;">Doubao</td>
+      <td>18.18</td><td>27.86</td><td>9.68</td>
+      <td>25.25</td><td>37.57</td><td>12.31</td>
+      <td>7.93</td><td>10.54</td><td>2.61</td>
+      <td>9.94</td><td>30.88</td><td>20.94</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;">ElevenLabs</td>
+      <td>19.58</td><td>37.95</td><td>18.36</td>
+      <td>25.70</td><td>36.69</td><td>10.99</td>
+      <td>8.50</td><td>11.34</td><td>2.85</td>
+      <td>11.49</td><td>17.85</td><td>6.37</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;">GPT-4o</td>
+      <td>-</td><td>-</td><td>-</td>
+      <td>-</td><td>-</td><td>-</td>
+      <td>-</td><td>-</td><td>-</td>
+      <td>14.37</td><td>23.67</td><td>9.31</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;">Gemini 2.5 Pro</td>
+      <td>42.70</td><td>53.42</td><td>10.72</td>
+      <td>27.43</td><td>41.64</td><td>14.21</td>
+      <td>7.38</td><td>10.23</td><td>2.85</td>
+      <td>15.46</td><td>24.15</td><td>8.69</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;">Gemini 3 Pro</td>
+      <td>22.75</td><td>27.43</td><td>4.68</td>
+      <td>26.75</td><td>32.84</td><td>6.09</td>
+      <td>-</td><td>-</td><td>-</td>
+      <td>8.62</td><td>14.73</td><td>6.11</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;">VIBEVOICE ASR</td>
+      <td>21.40</td><td>24.99</td><td>3.59</td>
+      <td>27.40</td><td>29.33</td><td>1.93</td>
+      <td>27.94</td><td>48.30</td><td>20.36</td>
+      <td>14.59</td><td>42.54</td><td>27.94</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;"><b>MOSS Transcribe Diarize</b></td>
+      <td>14.84</td><td>15.83</td><td>0.99</td>
+      <td>24.86</td><td>22.17</td><td>-2.69</td>
+      <td>5.97</td><td>7.37</td><td><b>1.40</b></td>
+      <td>6.36</td><td>12.76</td><td>6.40</td>
+    </tr>
+    <tr>
+      <td style="white-space: nowrap;"><b>MOSS Transcribe Diarize Pro</b></td>
+      <td><b>13.78</b></td><td><b>14.02</b></td><td><b>0.24</b></td>
+      <td><b>18.22</b></td><td><b>13.94</b></td><td><b>-4.27</b></td>
+      <td><b>4.46</b></td><td><b>6.97</b></td><td>2.51</td>
+      <td><b>5.86</b></td><td><b>11.78</b></td><td><b>5.92</b></td>
+    </tr>
+  </tbody>
+</table>
+</div>
 
 ## Quickstart
 
 ### Environment Setup
 
-We recommend Python 3.12 with a clean Conda environment. This repository uses custom Transformers model and processor code, so always load the model and processor with `trust_remote_code=True`.
+Use a clean Python environment. The model uses custom Transformers code, so load the model and processor with `trust_remote_code=True`.
 
 ```bash
-git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
-cd MOSS-Transcribe-Diarize
-
 conda create -n moss-transcribe-diarize python=3.12 -y
 conda activate moss-transcribe-diarize
 
-conda install -c conda-forge "ffmpeg=7" -y
-pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
-```
-
-Optional: if your GPU supports FlashAttention 2, install the optional runtime with:
-
-```bash
-pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
-```
-
-### Command Line Inference
-
-Run greedy decoding:
-
-```bash
-python infer.py \
-  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
-  --audio /path/to/audio_or_video.mp4 \
-  --decoding greedy \
-  --max-new-tokens 2048
-```
-
-Run sampling decoding:
+git clone https://github.com/OpenMOSS/MOSS-Transcribe-Diarize.git
+cd MOSS-Transcribe-Diarize
 
-```bash
-python infer.py \
-  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
-  --audio /path/to/audio_or_video.mp4 \
-  --decoding sample \
-  --temperature 0.7 \
-  --max-new-tokens 2048
+pip install torch torchaudio
+pip install -e .
 ```
 
-Return JSON output:
-
-```bash
-python infer.py \
-  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
-  --audio /path/to/audio_or_video.mp4 \
-  --json
-```
-
-Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.
+The GitHub package provides helper utilities such as audio/video loading, transcription message construction, transcript parsing, CLI inference, and the subtitle web app. The model weights and remote-code model files are loaded from this Hugging Face repository.
 
 ### Python Usage
 
@@ -162,6 +188,7 @@ Audio files are loaded through the Transformers audio loader. Video containers s
 import torch
 from transformers import AutoModelForCausalLM, AutoProcessor
 
+from moss_transcribe_diarize import parse_transcript
 from moss_transcribe_diarize.inference_utils import (
     build_transcription_messages,
     generate_transcription,
@@ -169,7 +196,7 @@ from moss_transcribe_diarize.inference_utils import (
 )
 
 model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
-audio_path = "/path/to/audio_or_video.mp4"
+audio_path = "audio.wav"
 
 device = resolve_device("auto")
 dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
@@ -183,7 +210,6 @@ model = AutoModelForCausalLM.from_pretrained(
 processor = AutoProcessor.from_pretrained(
     model_id,
     trust_remote_code=True,
-    fix_mistral_regex=True,
 )
 
 messages = build_transcription_messages(audio_path)
@@ -198,18 +224,91 @@ result = generate_transcription(
 )
 
 print(result["text"])
+
+for segment in parse_transcript(result["text"]):
+    print(segment.start, segment.end, segment.speaker, segment.text)
 ```
 
-The default prompt asks the model to output each speech segment with a start timestamp, a speaker label, transcript text, and an end timestamp. You can customize the instruction with:
+The message flow follows the common Qwen multimodal pattern:
 
-```python
-messages = build_transcription_messages(
-    audio_path,
-    prompt="Please transcribe the audio with timestamps and speaker labels.",
-)
+1. `processor.apply_chat_template(messages, tokenize=False)` renders text with audio placeholders.
+2. The helper utilities load audio waveforms from the same messages.
+3. `processor(text=text, audio=audios)` computes Whisper input features and expands audio placeholders.
+4. `model.generate(...)` produces timestamped transcription and diarization text.
+
+### Custom Prompt and Hotwords
+
+The default prompt is optimized for timestamped transcription and speaker diarization:
+
+```text
+请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。
+```
+
+To add hotwords, append a short hint to the default prompt:
+
+```text
+请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。热词提示：热词1, 热词2, 热词3
 ```
 
-The same can be done from the command line with `--prompt`.
+More prompt recipes are available in the GitHub repository: <https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/main/examples/prompts.md>
+
+### Serve with vLLM and SGLang
+
+MOSS-Transcribe-Diarize supports vLLM serving through the OpenAI-compatible transcription API:
+
+```bash
+pip install vllm
+vllm serve OpenMOSS-Team/MOSS-Transcribe-Diarize --trust-remote-code
+```
+
+```bash
+curl http://localhost:8000/v1/audio/transcriptions \
+  -F model="OpenMOSS-Team/MOSS-Transcribe-Diarize" \
+  -F file=@"audio.wav" \
+  -F response_format="json" \
+  -F temperature="0"
+```
+
+The same request format can be used with an SGLang OpenAI-compatible server:
+
+```bash
+python -m sglang.launch_server \
+  --model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --served-model-name OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+```bash
+curl http://localhost:30000/v1/audio/transcriptions \
+  -F model="OpenMOSS-Team/MOSS-Transcribe-Diarize" \
+  -F file=@"audio.wav" \
+  -F response_format="json" \
+  -F temperature="0"
+```
+
+### Subtitle Web App
+
+The source package includes a local subtitle workflow for upload, review, subtitle export, and optional FFmpeg burn-in:
+
+```bash
+mtd-subtitle-web \
+  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --host 127.0.0.1 \
+  --port 7860
+```
+
+Open `http://127.0.0.1:7860`, upload an audio/video file, review the parsed subtitle segments, then download JSON/SRT/ASS or burn an MP4 if `ffmpeg` and `ffprobe` are available on `PATH`.
+
+For batch processing:
+
+```bash
+mtd-subtitle /path/to/input.mp4 \
+  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --out-dir runs/example \
+  --render
+```
 
 ## Output Format
 
@@ -233,21 +332,26 @@ In this format:
 
 ## More Information
 
+* **GitHub**: <https://github.com/OpenMOSS/MOSS-Transcribe-Diarize>
 * **MOSI.AI**: <https://mosi.cn>
 * **OpenMOSS**: <https://www.open-moss.com>
 
-## LICENSE
+## License
 
 MOSS-Transcribe-Diarize is licensed under the Apache License 2.0.
 
 ## Citation
 
+If you use MOSS-Transcribe-Diarize, please cite the technical report:
+
 ```bibtex
-@misc{mosstranscribediarize2026,
-      title={MOSS-Transcribe-Diarize},
-      author={OpenMOSS Team},
-      year={2026},
-      howpublished={\url{https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize}},
-      note={Hugging Face model repository}
+@misc{moss_transcribe_diarize_2026,
+  title={MOSS Transcribe Diarize Technical Report},
+  author={{MOSI.AI}},
+  year={2026},
+  eprint={2601.01554},
+  archivePrefix={arXiv},
+  primaryClass={cs.SD},
+  url={https://arxiv.org/abs/2601.01554}
 }
 ```