---
license: other
license_name: longcat-audiodit-license
base_model: meituan-longcat/LongCat-AudioDiT-1B
tags:
  - audio
  - text-to-speech
  - tts
  - environmental-tts
  - flow-matching
  - dit
library_name: transformers
pipeline_tag: text-to-speech
---

# LongCat-AudioDiT Env-TTS — 5000-step fine-tune

Fine-tune of [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) for the
**three-stream env-tts task**: given a reference environment audio, a reference
speaker audio, and three text streams (env caption / speaker caption / target
speech text), generate target speech that places the target text in the
referenced environment with the referenced speaker timbre.

## Differences from the base model

The transformer adds **six learnable boundary tokens** (three latent-space, three
text-space):

```
latent sequence : [<boe>  z_env  <bos>  z_spk  <bon>  z_target]
text sequence   : [<boe_t> env_text_emb <bos_t> spk_text_emb <bon_t> target_text_emb]
```

`encode_multistream_text(env, spk, target, drop_env_text=…, drop_spk_text=…,
drop_target_text=…)` is the new entry-point. `AudioDiTModel.forward(...)` also
accepts a pre-assembled `prompt_latent` (replaces `prompt_audio`) so the inference
path can feed the boundary-tokenized three-stream prompt directly.

## Training summary

| Field | Value |
|---|---|
| Steps | 5000 |
| Effective batch | 16 × grad_accum 2 × 2 GPU = **64 rows / step** |
| Learning rate | cosine 5e-5 (warmup 250) |
| AdamW | β₁=0.9, β₂=0.999, wd=0.01 |
| EMA | disabled |
| LoRA | r=32, alpha=32, target = attn + ffn |
| Full-train | boundary tokens + AdaLN + text_conv + latent_embed + input_embed + output_proj + time_embed |
| Audio filter | target duration ∈ [3, 45] s |
| RMS normalize | three-stream independent to **-23 dBFS** (target_rms=0.0708) |
| Augmentation | noise + RIR on spk_audio (DNS5 64GB) |
| Data | [ChristianYang/Env-TTS-Clean](https://huggingface.co/datasets/ChristianYang/Env-TTS-Clean) |

## How to load

The model uses **custom code** in this repo, so pass `trust_remote_code=True`:

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "meituan-longcat/LongCat-AudioDiT-Env-TTS-1B-5000Step",
    trust_remote_code=True,
).cuda().eval()

tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder_model)
```

For end-to-end env-tts inference (three-stream prompt + ASR fallback for missing
env/spk text) see the training repo's `inference_env_tts.py`.

## License

Inherits the original [meituan-longcat/LongCat-AudioDiT-1B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B) license.