File size: 2,675 Bytes
720ba81 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
license: other
license_name: longcat-audiodit-license
base_model: meituan-longcat/LongCat-AudioDiT-1B
tags:
- audio
- text-to-speech
- tts
- environmental-tts
- flow-matching
- dit
library_name: transformers
pipeline_tag: text-to-speech
---
# LongCat-AudioDiT Env-TTS — 5000-step fine-tune
Fine-tune of [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) for the
**three-stream env-tts task**: given a reference environment audio, a reference
speaker audio, and three text streams (env caption / speaker caption / target
speech text), generate target speech that places the target text in the
referenced environment with the referenced speaker timbre.
## Differences from the base model
The transformer adds **six learnable boundary tokens** (three latent-space, three
text-space):
```
latent sequence : [<boe> z_env <bos> z_spk <bon> z_target]
text sequence : [<boe_t> env_text_emb <bos_t> spk_text_emb <bon_t> target_text_emb]
```
`encode_multistream_text(env, spk, target, drop_env_text=…, drop_spk_text=…,
drop_target_text=…)` is the new entry-point. `AudioDiTModel.forward(...)` also
accepts a pre-assembled `prompt_latent` (replaces `prompt_audio`) so the inference
path can feed the boundary-tokenized three-stream prompt directly.
## Training summary
| Field | Value |
|---|---|
| Steps | 5000 |
| Effective batch | 16 × grad_accum 2 × 2 GPU = **64 rows / step** |
| Learning rate | cosine 5e-5 (warmup 250) |
| AdamW | β₁=0.9, β₂=0.999, wd=0.01 |
| EMA | disabled |
| LoRA | r=32, alpha=32, target = attn + ffn |
| Full-train | boundary tokens + AdaLN + text_conv + latent_embed + input_embed + output_proj + time_embed |
| Audio filter | target duration ∈ [3, 45] s |
| RMS normalize | three-stream independent to **-23 dBFS** (target_rms=0.0708) |
| Augmentation | noise + RIR on spk_audio (DNS5 64GB) |
| Data | [ChristianYang/Env-TTS-Clean](https://huggingface.co/datasets/ChristianYang/Env-TTS-Clean) |
## How to load
The model uses **custom code** in this repo, so pass `trust_remote_code=True`:
```python
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained(
"meituan-longcat/LongCat-AudioDiT-Env-TTS-1B-5000Step",
trust_remote_code=True,
).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder_model)
```
For end-to-end env-tts inference (three-stream prompt + ASR fallback for missing
env/spk text) see the training repo's `inference_env_tts.py`.
## License
Inherits the original [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) license.
|