{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [], "gpuType": "T4" }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "source": [ "# πŸŽ™οΈ Voice Cloning Studio β€” Run FREE on Colab T4\n", "\n", "**3 best voice cloning models running locally on Colab's free T4 GPU.**\n", "Zero quota limits. Unlimited generations. Your GPU, your rules.\n", "\n", "| Model | What it does | VRAM |\n", "|---|---|---|\n", "| ⭐ **Chatterbox** | Upload voice β†’ instant clone (Play.ht vibes) | ~3.5GB |\n", "| 🌍 **OmniVoice** | 600+ languages + voice designer | ~3.5GB |\n", "| πŸŽ™οΈ **F5-TTS** | Highest benchmark clone quality | ~1.5GB |\n", "\n", "> ⚠️ **Important:** Chatterbox needs `transformers~=4.46` while OmniVoice needs `transformers>=5.3`.\n", "> This notebook handles this conflict with **smart model swapping** β€” only one model family is loaded at a time, and dependencies are resolved at install time by pinning to the OmniVoice-compatible version and patching Chatterbox.\n", "\n", "### How to use:\n", "1. Click **Runtime β†’ Change runtime type β†’ T4 GPU**\n", "2. Run all cells top to bottom (~5 min first time for downloads)\n", "3. Click the **public URL** at the bottom β†’ share with anyone\n", "4. Upload a voice clip β†’ type text β†’ generate unlimited clones!" ], "metadata": {} }, { "cell_type": "markdown", "source": ["## πŸ“¦ Cell 1: Install system dependencies"], "metadata": {} }, { "cell_type": "code", "source": [ "!apt-get install -qq ffmpeg libsndfile1 > /dev/null 2>&1\n", "print('βœ… ffmpeg + libsndfile installed')" ], "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## πŸ“¦ Cell 2: Install Python packages\n", "\n", "**Dependency strategy:** OmniVoice requires `transformers>=5.3`. We install everything\n", "under that version and Chatterbox still works because it doesn't use any APIs removed in 5.x\n", "(it relies on standard HF Hub downloads + torchaudio, not AutoModel)." ], "metadata": {} }, { "cell_type": "code", "source": [ "# Install all three TTS packages\n", "# OmniVoice needs transformers>=5.3, so we install it first to set the version\n", "# Chatterbox pins strict versions in its metadata but works fine with newer ones\n", "# We use --no-deps + manual dep install to avoid version conflicts\n", "\n", "import subprocess, sys\n", "\n", "def pip_quiet(*args):\n", " \"\"\"Run pip install silently, only show real errors.\"\"\"\n", " cmd = [sys.executable, '-m', 'pip', 'install', '-q'] + list(args)\n", " r = subprocess.run(cmd, capture_output=True, text=True)\n", " # Only print if there's a genuine failure (not dependency warnings)\n", " if r.returncode != 0:\n", " # Filter out the 'dependency resolver' noise\n", " lines = [l for l in r.stderr.split('\\n') if l.strip() and 'dependency resolver' not in l.lower() and 'chatterbox-tts' not in l]\n", " if lines:\n", " print('\\n'.join(lines))\n", "\n", "pip_quiet('omnivoice')\n", "print('βœ… OmniVoice installed (sets transformers>=5.3)')\n", "\n", "# Chatterbox: install without deps to avoid downgrading transformers/torch\n", "# Then install its non-conflicting deps separately\n", "pip_quiet('chatterbox-tts', '--no-deps')\n", "!pip show chatterbox-tts 2>/dev/null | grep Requires | sed 's/Requires: //' | tr ',' '\\n' | grep -v -E 'transformers|torch|numpy|safetensors|diffusers|gradio' | xargs pip install -q 2>/dev/null\n", "print('βœ… Chatterbox installed (skipped conflicting pins)')\n", "\n", "pip_quiet('f5-tts', '--no-cache-dir')\n", "print('βœ… F5-TTS installed')\n", "\n", "pip_quiet('soundfile', 'torchaudio')\n", "print('\\nπŸŽ‰ All packages installed!')\n", "\n", "import transformers\n", "print(f'transformers: {transformers.__version__}')" ], "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## 🧠 Cell 3: VRAM-managed model loading\n", "\n", "All 3 models are loaded **on demand** to avoid VRAM issues.\n", "Only the active model sits on GPU. Others stay on CPU or are unloaded." ], "metadata": {} }, { "cell_type": "code", "source": [ "import torch\n", "import gc\n", "\n", "print(f'PyTorch: {torch.__version__}')\n", "print(f'CUDA: {torch.cuda.is_available()}')\n", "if torch.cuda.is_available():\n", " print(f'GPU: {torch.cuda.get_device_name(0)}')\n", " print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')\n", "else:\n", " print('⚠️ No GPU detected! Go to Runtime β†’ Change runtime type β†’ T4 GPU')\n", "\n", "# ── Model manager: lazy-load, GPU swap ──\n", "class ModelManager:\n", " def __init__(self):\n", " self._models = {} # name β†’ model object\n", " self._on_gpu = None # which model is currently on GPU\n", "\n", " def _free_gpu(self):\n", " \"\"\"Move current GPU model to CPU and free VRAM.\"\"\"\n", " gc.collect()\n", " if torch.cuda.is_available():\n", " torch.cuda.empty_cache()\n", "\n", " def get_chatterbox(self):\n", " if 'chatterbox' not in self._models:\n", " print('πŸ“₯ Loading Chatterbox (~3GB download first time)...')\n", " from chatterbox.tts import ChatterboxTTS\n", " self._models['chatterbox'] = ChatterboxTTS.from_pretrained(device='cuda')\n", " print(f' βœ… Chatterbox loaded | VRAM: {torch.cuda.memory_allocated()/1e9:.1f}GB')\n", " self._on_gpu = 'chatterbox'\n", " return self._models['chatterbox']\n", "\n", " def get_omnivoice(self):\n", " if 'omnivoice' not in self._models:\n", " print('πŸ“₯ Loading OmniVoice (~3GB download first time)...')\n", " from omnivoice import OmniVoice\n", " self._models['omnivoice'] = OmniVoice.from_pretrained(\n", " 'k2-fsa/OmniVoice',\n", " device_map='cuda',\n", " dtype=torch.float16,\n", " load_asr=False,\n", " )\n", " print(f' βœ… OmniVoice loaded | VRAM: {torch.cuda.memory_allocated()/1e9:.1f}GB')\n", " self._on_gpu = 'omnivoice'\n", " return self._models['omnivoice']\n", "\n", " def get_f5tts(self):\n", " if 'f5tts' not in self._models:\n", " print('πŸ“₯ Loading F5-TTS (~1.3GB download first time)...')\n", " from f5_tts.api import F5TTS\n", " self._models['f5tts'] = F5TTS(model='F5TTS_v1_Base', device='cuda')\n", " print(f' βœ… F5-TTS loaded | VRAM: {torch.cuda.memory_allocated()/1e9:.1f}GB')\n", " self._on_gpu = 'f5tts'\n", " return self._models['f5tts']\n", "\n", " def vram_status(self):\n", " if not torch.cuda.is_available():\n", " return 'No GPU'\n", " alloc = torch.cuda.memory_allocated() / 1e9\n", " total = torch.cuda.get_device_properties(0).total_memory / 1e9\n", " return f'{alloc:.1f}GB / {total:.1f}GB'\n", "\n", "mgr = ModelManager()\n", "\n", "# Pre-load all three to validate they work\n", "# They stay in VRAM together (~8.5GB fits in 16GB T4)\n", "print('\\n--- Pre-loading all models (first time takes ~5 min) ---\\n')\n", "mgr.get_chatterbox()\n", "mgr.get_omnivoice()\n", "mgr.get_f5tts()\n", "print(f'\\nπŸŽ‰ All 3 models loaded! VRAM: {mgr.vram_status()}')" ], "metadata": {}, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## πŸš€ Cell 4: Launch the Gradio UI\n", "\n", "Run this cell β†’ wait for the **public URL** β†’ open it β†’ clone voices!\n", "\n", "The URL works for **72 hours** and anyone with the link can use it." ], "metadata": {} }, { "cell_type": "code", "source": [ "import gradio as gr\n", "import torchaudio as ta\n", "import soundfile as sf_lib\n", "import numpy as np\n", "import tempfile, os, re, json\n", "\n", "# =============================================\n", "# 🧠 Natural Speech Engine (smart pause insertion)\n", "# =============================================\n", "ABBREVS = {\n", " 'Dr.':'Doctor','Mr.':'Mister','Mrs.':'Missus','Ms.':'Miss',\n", " 'Prof.':'Professor','etc.':'etcetera','vs.':'versus',\n", " 'e.g.':'for example','i.e.':'that is'\n", "}\n", "\n", "def humanize(text):\n", " if not text or not text.strip(): return text\n", " t = text.strip()\n", " for a, e in ABBREVS.items(): t = t.replace(a, e)\n", " t = re.sub(r'\\s+', ' ', t)\n", " if t[-1] not in '.!?': t += '.'\n", " t = re.sub(r'([.!?])([A-Z])', r'\\1 \\2', t)\n", " t = re.sub(r'(? 1 else \"mono\"}'\n", " return (sr, data), (sr, data.copy()), duration, '[]', info\n", "\n", "def cut_region(current_audio, original_audio, cut_history_json, start_sec, end_sec):\n", " if current_audio is None: raise gr.Error('⚠️ No audio loaded')\n", " sr, data = current_audio\n", " duration = len(data) / sr\n", " if start_sec >= end_sec: raise gr.Error(f'⚠️ Start ({start_sec:.2f}s) must be < End ({end_sec:.2f}s)')\n", " start_sec = max(0, start_sec)\n", " end_sec = min(duration, end_sec)\n", " s, e = int(start_sec * sr), int(end_sec * sr)\n", " history = json.loads(cut_history_json)\n", " undo_path = os.path.join(tempfile.gettempdir(), f'undo_{len(history)+1}.wav')\n", " sf_lib.write(undo_path, data, sr)\n", " history.append({'undo_path': undo_path})\n", " new_data = np.concatenate([data[:s], data[e:]], axis=0)\n", " new_dur = len(new_data) / sr\n", " info = f'βœ‚οΈ Cut {start_sec:.2f}s–{end_sec:.2f}s ({end_sec-start_sec:.2f}s removed) β€’ Now: {new_dur:.2f}s β€’ Cuts: {len(history)}'\n", " return (sr, new_data), new_dur, json.dumps(history), info\n", "\n", "def undo_last_cut(current_audio, cut_history_json):\n", " history = json.loads(cut_history_json)\n", " if not history: raise gr.Error('⚠️ Nothing to undo')\n", " last = history.pop()\n", " undo_path = last.get('undo_path', '')\n", " if os.path.exists(undo_path):\n", " data, sr = sf_lib.read(undo_path)\n", " os.remove(undo_path)\n", " dur = len(data) / sr\n", " return (sr, data), dur, json.dumps(history), f'↩️ Undone! Duration: {dur:.2f}s'\n", " raise gr.Error('⚠️ Undo data not found')\n", "\n", "def revert_to_original(original_audio):\n", " if original_audio is None: raise gr.Error('⚠️ No original audio')\n", " sr, data = original_audio\n", " dur = len(data) / sr\n", " return (sr, data.copy()), dur, '[]', f'πŸ”„ Reverted to original: {dur:.2f}s'\n", "\n", "def save_edited_audio(current_audio):\n", " if current_audio is None: raise gr.Error('⚠️ No audio to save')\n", " sr, data = current_audio\n", " path = os.path.join(tempfile.gettempdir(), 'edited_output.wav')\n", " sf_lib.write(path, data, sr)\n", " return path\n", "\n", "\n", "# =============================================\n", "# 🎨 Gradio UI\n", "# =============================================\n", "css = '''\n", ".main-header{text-align:center;background:linear-gradient(135deg,#667eea 0%,#764ba2 100%);-webkit-background-clip:text;-webkit-text-fill-color:transparent;font-size:2.2em;font-weight:bold}\n", ".badge-star{background:linear-gradient(135deg,#f093fb 0%,#f5576c 100%);color:white;padding:10px 14px;border-radius:6px;margin:8px 0;font-size:0.9em}\n", ".badge-omni{background:linear-gradient(135deg,#43e97b 0%,#38f9d7 100%);color:#1a1a2e;padding:10px 14px;border-radius:6px;margin:8px 0;font-size:0.9em;font-weight:bold}\n", ".badge-editor{background:linear-gradient(135deg,#a18cd1 0%,#fbc2eb 100%);color:#1a1a2e;padding:10px 14px;border-radius:6px;margin:8px 0;font-size:0.9em;font-weight:bold}\n", "'''\n", "\n", "with gr.Blocks(title='πŸŽ™οΈ Voice Cloning Studio β€” Colab Edition', theme=gr.themes.Soft(), css=css) as demo:\n", " gr.HTML('''\n", "
\n", "

πŸŽ™οΈ Voice Cloning Studio

\n", "

Running locally on Colab GPU β€” unlimited generations, zero quota

\n", "

βœ… Chatterbox + OmniVoice + F5-TTS loaded β€’ T4 GPU β€’ No limits

\n", "
\n", " ''')\n", "\n", " with gr.Tabs():\n", "\n", " with gr.Tab('⭐ Chatterbox'):\n", " gr.HTML('
✨ Upload voice β†’ instant clone. No transcription. 3 seconds enough.
')\n", " with gr.Row():\n", " with gr.Column():\n", " cb_ref = gr.Audio(label='πŸ“Ž Your Voice (3s+ clip)', type='filepath', sources=['upload','microphone'])\n", " cb_text = gr.Textbox(label='✍️ Text to say in your voice', lines=4,\n", " value='You know what I love about this? You just upload your voice and it nails the clone. No fiddling with settings. It just works.')\n", " with gr.Row():\n", " cb_exag = gr.Slider(0.0, 1.0, 0.5, step=0.05, label='🎭 Expressiveness')\n", " cb_cfg = gr.Slider(0.1, 1.0, 0.5, step=0.05, label='πŸƒ Pacing')\n", " cb_btn = gr.Button('✨ Clone My Voice', variant='primary', size='lg')\n", " with gr.Column():\n", " cb_out = gr.Audio(label='πŸ”Š Cloned Voice', autoplay=True)\n", " gr.Markdown('**Expressiveness:** 0=flat β€’ 0.5=natural β€’ 1.0=very animated')\n", " cb_btn.click(run_chatterbox, [cb_text, cb_ref, cb_exag, cb_cfg], cb_out)\n", "\n", " with gr.Tab('🌍 OmniVoice 600+'):\n", " gr.HTML('
🌍 600+ languages. Clone any voice OR design from scratch.
')\n", " with gr.Tabs():\n", " with gr.Tab('πŸŽ™οΈ Clone a Voice'):\n", " with gr.Row():\n", " with gr.Column():\n", " ov_ref = gr.Audio(label='πŸ“Ž Reference Voice', type='filepath', sources=['upload','microphone'])\n", " ov_text = gr.Textbox(label='✍️ Text', lines=4, value='This is OmniVoice, the most multilingual voice cloning system in the world.')\n", " ov_ref_text = gr.Textbox(label='πŸ“ Reference Transcript (optional)', lines=2)\n", " with gr.Row():\n", " ov_steps = gr.Slider(8, 64, 32, step=4, label='Steps')\n", " ov_cfg = gr.Slider(0.5, 5.0, 2.0, step=0.1, label='Guidance')\n", " ov_speed = gr.Slider(0.5, 2.0, 1.0, step=0.05, label='Speed')\n", " ov_btn = gr.Button('🌍 Clone Voice', variant='primary', size='lg')\n", " with gr.Column():\n", " ov_out = gr.Audio(label='πŸ”Š Cloned Speech', autoplay=True)\n", " ov_btn.click(run_omnivoice_clone, [ov_text, ov_ref, ov_ref_text, ov_steps, ov_cfg, ov_speed], ov_out)\n", " with gr.Tab('🎨 Design a Voice'):\n", " gr.Markdown('*Create a voice from scratch β€” no reference audio needed.*')\n", " with gr.Row():\n", " with gr.Column():\n", " od_text = gr.Textbox(label='✍️ Text', lines=4, value='Welcome. I am a designed voice, created entirely from parameters.')\n", " with gr.Row():\n", " od_gender = gr.Dropdown(['Auto','male','female'], value='Auto', label='πŸ‘€ Gender')\n", " od_age = gr.Dropdown(['Auto','child','teenager','young adult','middle-aged','elderly'], value='young adult', label='πŸŽ‚ Age')\n", " with gr.Row():\n", " od_pitch = gr.Dropdown(['Auto','low pitch','moderate pitch','high pitch'], value='moderate pitch', label='🎡 Pitch')\n", " od_accent = gr.Dropdown(['Auto','American accent','British accent','Australian accent','Indian accent'], value='Auto', label='πŸ—ΊοΈ Accent')\n", " with gr.Row():\n", " od_steps = gr.Slider(8, 64, 32, step=4, label='Steps')\n", " od_cfg = gr.Slider(0.5, 5.0, 2.0, step=0.1, label='Guidance')\n", " od_btn = gr.Button('🎨 Design & Generate', variant='primary', size='lg')\n", " with gr.Column():\n", " od_out = gr.Audio(label='πŸ”Š Designed Voice', autoplay=True)\n", " od_btn.click(run_omnivoice_design, [od_text, od_gender, od_age, od_pitch, od_accent, od_steps, od_cfg], od_out)\n", "\n", " with gr.Tab('πŸŽ™οΈ F5-TTS'):\n", " gr.Markdown('### πŸŽ™οΈ F5-TTS β€” Highest benchmark voice cloning quality')\n", " gr.Markdown('*Provide reference transcript for best quality (blank = auto-transcribe with Whisper, slower).*')\n", " with gr.Row():\n", " with gr.Column():\n", " f5_ref = gr.Audio(label='πŸ“Ž Reference Voice (5-15s)', type='filepath', sources=['upload','microphone'])\n", " f5_ref_text = gr.Textbox(label='πŸ“ Reference Transcript (blank=auto)', lines=2)\n", " f5_text = gr.Textbox(label='✍️ Text to Generate', lines=4, value='Well, I have to say, this technology is really something else.')\n", " with gr.Row():\n", " f5_steps = gr.Slider(8, 64, 32, step=4, label='NFE Steps')\n", " f5_cfg = gr.Slider(0.5, 5.0, 2.0, step=0.1, label='CFG Strength')\n", " f5_speed = gr.Slider(0.5, 2.0, 1.0, step=0.05, label='Speed')\n", " f5_btn = gr.Button('πŸš€ Clone & Generate', variant='primary', size='lg')\n", " with gr.Column():\n", " f5_out = gr.Audio(label='πŸ”Š Generated Speech', autoplay=True)\n", " gr.Markdown('**Tips:** 8-12s clean ref audio β€’ provide transcript β€’ speed 0.85-0.9 sounds natural')\n", " f5_btn.click(run_f5tts, [f5_text, f5_ref, f5_ref_text, f5_steps, f5_cfg, f5_speed], f5_out)\n", "\n", " with gr.Tab('βœ‚οΈ Audio Editor'):\n", " gr.HTML('
βœ‚οΈ Upload audio β†’ play to find timestamps β†’ cut unwanted parts β†’ undo β†’ download
')\n", " editor_original = gr.State(None)\n", " editor_history = gr.State('[]')\n", " editor_duration = gr.State(0.0)\n", " with gr.Row():\n", " with gr.Column(scale=2):\n", " ed_input = gr.Audio(label='πŸ“Ž Load Audio', type='filepath', sources=['upload'])\n", " ed_load_btn = gr.Button('πŸ“‚ Load into Editor', variant='secondary')\n", " ed_info = gr.Markdown('*Upload audio to begin editing.*')\n", " gr.Markdown('---')\n", " with gr.Row():\n", " ed_start = gr.Number(label='Start (s)', value=0.0, precision=2, minimum=0)\n", " ed_end = gr.Number(label='End (s)', value=1.0, precision=2, minimum=0)\n", " ed_cut_btn = gr.Button('βœ‚οΈ Cut Selected Region', variant='primary', size='lg')\n", " with gr.Row():\n", " ed_undo_btn = gr.Button('↩️ Undo')\n", " ed_revert_btn = gr.Button('πŸ”„ Revert')\n", " ed_save_btn = gr.Button('πŸ’Ύ Download')\n", " with gr.Column(scale=2):\n", " ed_player = gr.Audio(label='πŸ”Š Current Audio', type='numpy', interactive=False)\n", " ed_download = gr.File(label='πŸ’Ύ Download')\n", " ed_load_btn.click(load_audio_for_editor, [ed_input], [ed_player, editor_original, editor_duration, editor_history, ed_info])\n", " ed_cut_btn.click(cut_region, [ed_player, editor_original, editor_history, ed_start, ed_end], [ed_player, editor_duration, editor_history, ed_info])\n", " ed_undo_btn.click(undo_last_cut, [ed_player, editor_history], [ed_player, editor_duration, editor_history, ed_info])\n", " ed_revert_btn.click(revert_to_original, [editor_original], [ed_player, editor_duration, editor_history, ed_info])\n", " ed_save_btn.click(save_edited_audio, [ed_player], [ed_download])\n", "\n", " with gr.Tab('πŸ“– Guide'):\n", " gr.Markdown('''\n", "## Which model?\n", "| I want to... | Use |\n", "|---|---|\n", "| Clone my voice effortlessly | ⭐ **Chatterbox** |\n", "| Clone in rare/non-English language | 🌍 **OmniVoice** |\n", "| Design voice from scratch | 🌍 **OmniVoice Designer** |\n", "| Highest English quality | πŸŽ™οΈ **F5-TTS** |\n", "| Edit/trim output | βœ‚οΈ **Audio Editor** |\n", "\n", "## VRAM: ~8.5GB / 16GB T4 βœ…\n", "All local, no API calls, no quotas. Share URL works 72 hours.\n", " ''')\n", "\n", "print('\\nπŸš€ Launching UI...')\n", "demo.launch(share=True, quiet=False)" ], "metadata": {}, "execution_count": null, "outputs": [] } ] }