README.md

---
library_name: transformers
tags:
- tts
- audio
- text-to-speech
- speech-generation
- voice-cloning
- de
pipeline_tag: text-to-speech
license: cc-by-nc-sa-4.0
language:
- de
base_model:
- YatharthS/MiraTTS
---

# 🚀🥔 MiraToffel (German MiraTTS)
<img src="https://huggingface.co/SebastianBodza/MiraToffel_miraTTS_german/resolve/main/img.webp" alt="MiraToffel" style="width:256px"/>

**MiraToffel** is a german finetune of the MiraTTS model. It generates clear, realistic speech at speeds up to **100x realtime** (using batching/Lmdeploy) while maintaining high-fidelity **48kHz audio output**.  
Designed to be memory efficient, it runs comfortably within **6GB VRAM** with latencies as low as **100ms**.  
<div style="display: flex;align-items: center; gap: 12px">
<a target="_blank" href="https://huggingface.co/spaces/SebastianBodza/Kartoffel-MiraToffel-TTS">
<img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>
</a>
</div>  

## Key Benefits

- **⚡ Incredibly Fast:** Capable of speeds over 100x realtime.
- **🎧 High Quality:** Generates crisp **48kHz** audio, surpassing standard 24kHz/32kHz models.
- **💾 Memory Efficient:** Fully functional on consumer GPUs with <6GB VRAM.
- **🗣️ Voice Cloning:** Zero-shot voice cloning capabilities with high resemblance to reference audio.

## What Kartoffelmodel to use? 
- Use the Kartoffelbox for a better voice cloning
- Use MiraTTS for a higher quality speech, pronounciation, ... , but worse voice cloning
- Kartoffelbox-Turbo is just an experimental model

## Which Kartoffel-Model should I use?

Since there are multiple versions of the "Kartoffel" fine-tunes, here is a quick guide to help you choose the right one for your needs:

| Model | Best Used For | Pros | Cons |
| :--- | :--- | :--- | :--- |
| **[Kartoffelbox](https://huggingface.co/SebastianBodza/Kartoffelbox-v0.1)** | **Voice Cloning** | • Superior voice cloning accuracy<br>• Good emotion control | • Lower stability & worse pronounciation than Mira |
| **[MiraToffel](https://huggingface.co/SebastianBodza/MiraToffel_miraTTS_german/)** | **High Fidelity Speech** | • Better pronunciation & stability<br>• Fast generation | • Voice cloning is less accurate than Kartoffelbox |
| **[Kartoffelbox-Turbo](https://huggingface.co/SebastianBodza/Kartoffelbox_Turbo)** | **Experimentation** | • Faster than base Kartoffelbox | • **Experimental status**<br>• unstable |

**Summary:**
* Choose **Kartoffelbox** if your priority is sounding *close* to the reference speaker.
* Choose **MiraToffel** if your priority is clear audio quality and better German pronunciation, even if the voice match isn't 100% perfect.


## Installation
You can install the optimized Mira library directly from GitHub:

```bash
uv pip install git+https://github.com/ysharma3501/MiraTTS.git
# Or using standard pip
pip install git+https://github.com/ysharma3501/MiraTTS.git
```

## Usage
### Standard Usage with lmdeploy

```python
from mira.model import MiraTTS
from IPython.display import Audio

# Load the German Fine-tune
mira_tts = MiraTTS('SebastianBodza/MiraToffel_miraTTS_german') 
reference_file = "german_reference.wav" 

text = "Na, hast du schon mal von MiraToffel gehört? Das ist echt der Wahnsinn!"

# 1. Encode the reference voice
context_tokens = mira_tts.encode_audio(reference_file)

# 2. Generate Speech
audio = mira_tts.generate(text, context_tokens)

# 3. Play/Save (Audio is 48kHz)
Audio(audio, rate=48000)
```

**Batching Example:**

```python
text_batch = [
    "Hallo! Wie geht es dir heute?", 
    "Ich finde diese Technologie faszinierend."
]

context_tokens = [mira_tts.encode_audio(reference_file)]

# Generate multiple sentences at once
audio_batch = mira_tts.batch_generate(text_batch, context_tokens)
```

### Usage with transformers
If you need granular control or want to integrate into an existing `transformers` pipeline without the Mira wrapper, use the following script.
```python
import torch
import numpy as np
import librosa
import soundfile as sf
from transformers import AutoTokenizer, AutoModelForCausalLM
import logging

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
REPO_ID = "SebastianBodza/MiraToffel_miraTTS_german" 

def _init_model(repo_id):
    """Initializes a model using Standard Transformers."""
    logging.info(f"🚀 Initializing Transformers Model from {repo_id}...")
    tokenizer = AutoTokenizer.from_pretrained(repo_id)
    model = AutoModelForCausalLM.from_pretrained(repo_id)
    model.to(DEVICE)    
    return {"model": model, "tokenizer": tokenizer}

def generate_mira_speech(model_obj, text, audio_path):
    model = model_obj["model"]
    tokenizer = model_obj["tokenizer"]   

    # 1. Load and encode reference audio
    audio_array, sr = librosa.load(audio_path, sr=16000)
    context_tokens = tts_codec.encode(audio_array)   

    # 2. Format prompt
    formatted_prompt = tts_codec.format_prompt(text, context_tokens, None)   

    # 3. Tokenize inputs
    model_inputs = tokenizer([formatted_prompt], return_tensors="pt").to(DEVICE)   

    # 4. Generate
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        top_p=1.0,
        repetition_penalty=1.2,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id
    )  

    # 5. Decode output
    generated_ids_trimmed = generated_ids[:, model_inputs.input_ids.shape[1]:]
    predicts_text = tokenizer.batch_decode(generated_ids_trimmed, skip_special_tokens=False)[0]
    
    # 6. Convert back to audio
    audio = tts_codec.decode(predicts_text, context_tokens)

    if isinstance(audio, torch.Tensor):
        audio = audio.detach().cpu().float().numpy()
    elif isinstance(audio, np.ndarray):
        audio = audio.astype(np.float32)     
    return audio, 48000

# --- Execution ---
print("Loading MiraToffel...")
model_data = _init_model(REPO_ID)

text_input = "Hallo, das ist ein Test des MiraToffel Systems."
reference_audio = "path/to/german_reference.wav"

print("Generating...")
audio_out, sample_rate = generate_mira_speech(model_data, text_input, reference_audio)

sf.write("miratoffel_output.wav", audio_out, sample_rate)

print("Saved to miratoffel_output.wav")
```


## Acknowledgements
* Thanks to **YatharthS** for the training and the finetuning code of miraTTS
* Check out the [GitHub Repository](https://github.com/ysharma3501/MiraTTS) for finetuning notebooks and further details.