| --- |
| license: other |
| license_name: longcat-audiodit-license |
| base_model: meituan-longcat/LongCat-AudioDiT-1B |
| tags: |
| - audio |
| - text-to-speech |
| - tts |
| - environmental-tts |
| - flow-matching |
| - dit |
| library_name: transformers |
| pipeline_tag: text-to-speech |
| --- |
| |
| # LongCat-AudioDiT Env-TTS — 5000-step fine-tune |
|
|
| Fine-tune of [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) for the |
| **three-stream env-tts task**: given a reference environment audio, a reference |
| speaker audio, and three text streams (env caption / speaker caption / target |
| speech text), generate target speech that places the target text in the |
| referenced environment with the referenced speaker timbre. |
|
|
| ## Differences from the base model |
|
|
| The transformer adds **six learnable boundary tokens** (three latent-space, three |
| text-space): |
|
|
| ``` |
| latent sequence : [<boe> z_env <bos> z_spk <bon> z_target] |
| text sequence : [<boe_t> env_text_emb <bos_t> spk_text_emb <bon_t> target_text_emb] |
| ``` |
|
|
| `encode_multistream_text(env, spk, target, drop_env_text=…, drop_spk_text=…, |
| drop_target_text=…)` is the new entry-point. `AudioDiTModel.forward(...)` also |
| accepts a pre-assembled `prompt_latent` (replaces `prompt_audio`) so the inference |
| path can feed the boundary-tokenized three-stream prompt directly. |
|
|
| ## Training summary |
|
|
| | Field | Value | |
| |---|---| |
| | Steps | 5000 | |
| | Effective batch | 16 × grad_accum 2 × 2 GPU = **64 rows / step** | |
| | Learning rate | cosine 5e-5 (warmup 250) | |
| | AdamW | β₁=0.9, β₂=0.999, wd=0.01 | |
| | EMA | disabled | |
| | LoRA | r=32, alpha=32, target = attn + ffn | |
| | Full-train | boundary tokens + AdaLN + text_conv + latent_embed + input_embed + output_proj + time_embed | |
| | Audio filter | target duration ∈ [3, 45] s | |
| | RMS normalize | three-stream independent to **-23 dBFS** (target_rms=0.0708) | |
| | Augmentation | noise + RIR on spk_audio (DNS5 64GB) | |
| | Data | [ChristianYang/Env-TTS-Clean](https://huggingface.co/datasets/ChristianYang/Env-TTS-Clean) | |
|
|
| ## How to load |
|
|
| The model uses **custom code** in this repo, so pass `trust_remote_code=True`: |
|
|
| ```python |
| from transformers import AutoModel, AutoTokenizer |
| |
| model = AutoModel.from_pretrained( |
| "meituan-longcat/LongCat-AudioDiT-Env-TTS-1B-5000Step", |
| trust_remote_code=True, |
| ).cuda().eval() |
| |
| tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder_model) |
| ``` |
|
|
| For end-to-end env-tts inference (three-stream prompt + ASR fallback for missing |
| env/spk text) see the training repo's `inference_env_tts.py`. |
|
|
| ## License |
|
|
| Inherits the original [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) license. |
|
|
