Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,17 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+cpu_fp16/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+cpu_fp16/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+cpu_fp32/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+cpu_fp32/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+cpu_int4/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+cpu_int4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+gpu_fp16/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+gpu_fp16/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+gpu_fp32/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+gpu_fp32/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+gpu_int4/llm_decoder.onnx.data filter=lfs diff=lfs merge=lfs -text
+gpu_int4/tokenizer.json filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,8 @@
+# Downloaded checkpoint + intermediate/build artifacts
+model/
+cache/
+cpu_int4/models/*.onnx
+cpu_int4/models/*.onnx.data
+_genai_*/
+__pycache__/
+*.pyc

README.md

@@ -0,0 +1,92 @@
+---
+language:
+- en
+- zh
+license: mit
+pipeline_tag: text-to-speech
+tags:
+- Podcast
+- onnx
+- onnxruntime
+library_name: onnxruntime
+base_model:
+- microsoft/VibeVoice-1.5B
+---
+
+## VibeVoice: A Frontier Open-Source Text-to-Speech Model
+
+VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking.
+
+A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.
+
+The model can synthesize speech up to **90 minutes** long with up to **4 distinct speakers**, surpassing the typical 1-2 speaker limits of many prior models. 
+
+➡️ **Technical Report:** [VibeVoice Technical Report](https://arxiv.org/abs/2508.19205)
+
+➡️ **Project Page:** [microsoft/VibeVoice](https://microsoft.github.io/VibeVoice)
+
+➡️ **Code:** [microsoft/VibeVoice-Code](https://github.com/microsoft/VibeVoice)
+
+<p align="left">
+  <img src="figures/Fig1.png" alt="VibeVoice Overview" height="250px">
+</p>
+
+## Training Details
+Transformer-based Large Language Model (LLM) integrated with specialized acoustic and semantic tokenizers and a diffusion-based decoding head.
+- LLM: [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) for this release.
+- Tokenizers:
+    - Acoustic Tokenizer: Based on a σ-VAE variant (proposed in [LatentLM](https://arxiv.org/pdf/2412.08635)), with a mirror-symmetric encoder-decoder structure featuring 7 stages of modified Transformer blocks. Achieves 3200x downsampling from 24kHz input. Encoder/decoder components are ~340M parameters each.
+    - Semantic Tokenizer: Encoder mirrors the Acoustic Tokenizer's architecture (without VAE components). Trained with an ASR proxy task.
+- Diffusion Head: Lightweight module (4 layers, ~123M parameters) conditioned on LLM hidden states. Predicts acoustic VAE features using a Denoising Diffusion Probabilistic Models (DDPM) process. Uses Classifier-Free Guidance (CFG) and DPM-Solver (and variants) during inference.
+- Context Length: Trained with a curriculum increasing up to 65,536 tokens.
+- Training Stages:
+    - Tokenizer Pre-training: Acoustic and Semantic tokenizers are pre-trained separately.
+    - VibeVoice Training: Pre-trained tokenizers are frozen; only the LLM and diffusion head parameters are trained. A curriculum learning strategy is used for input sequence length (4k -> 16K -> 32K -> 64K). Text tokenizer not explicitly specified, but the LLM (Qwen2.5) typically uses its own. Audio is "tokenized" via the acoustic and semantic tokenizers.
+
+
+## Models
+| Model | Context Length | Generation Length |  Weight |
+|-------|----------------|----------|----------|
+| VibeVoice-0.5B-Streaming | - | - | [HF link](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B) |
+| VibeVoice-1.5B | 64K | ~90 min | You are here. |
+| VibeVoice-Large| 32K | ~45 min | Disabled |
+
+## Installation and Usage
+
+Please refer to [GitHub README](https://github.com/microsoft/VibeVoice?tab=readme-ov-file#installation)
+
+## Responsible Usage
+### Direct intended uses
+The VibeVoice model is limited to research purpose use exploring highly realistic audio dialogue generation detailed in the [tech report](https://arxiv.org/pdf/2508.19205). 
+
+### Out-of-scope uses
+Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by MIT License. Use to generate any text transcript. Furthermore, this release is not intended or licensed for any of the following scenarios:
+
+- Voice impersonation without explicit, recorded consent – cloning a real individual’s voice for satire, advertising, ransom, social‑engineering, or authentication bypass.
+- Disinformation or impersonation – creating audio presented as genuine recordings of real people or events.
+- Real‑time or low‑latency voice conversion – telephone or video‑conference “live deep‑fake” applications.
+- Unsupported language – the model is trained only on English and Chinese data; outputs in other languages are unsupported and may be unintelligible or offensive.
+- Generation of background ambience, Foley, or music – VibeVoice is speech‑only and will not produce coherent non‑speech audio.
+
+
+## Risks and limitations
+While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). 
+Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content.
+English and Chinese only: Transcripts in language other than English or Chinese may result in unexpected audio outputs.
+Non-Speech Audio: The model focuses solely on speech synthesis and does not handle background noise, music, or other sound effects.
+Overlapping Speech: The current model does not explicitly model or generate overlapping speech segments in conversations.
+
+
+## Recommendations
+We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.
+
+To mitigate the risks of misuse, we have:
+Embedded an audible disclaimer (e.g. “This segment was generated by AI”) automatically into every synthesized audio file.
+Added an imperceptible watermark to generated audio so third parties can verify VibeVoice provenance. Please see contact information at the end of this model card.
+Logged inference requests (hashed) for abuse pattern detection and publishing aggregated statistics quarterly.
+Users are responsible for sourcing their datasets legally and ethically. This may include securing appropriate rights and/or anonymizing data prior to use with VibeVoice. Users are reminded to be mindful of data privacy concerns. 
+
+
+## Contact
+This project was conducted by members of Microsoft Research. We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at VibeVoice@microsoft.com.
+If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations.

common.py

@@ -0,0 +1,312 @@
+"""Shared building blocks for VibeVoice ONNX inference (TTS / ASR / Realtime).
+
+The three drivers (inference.py, inference_asr.py, inference_realtime.py) all lean on the
+same primitives, factored here:
+
+  * resolve()            — model key + checkpoint src + ONNX dir (reuses optimize.py's registry)
+  * OnnxLLM              — drives the genai-exported llm_decoder.onnx directly via onnxruntime,
+                           managing the 28×(k,v) KV cache + growing attention_mask. Works for
+                           BOTH the TTS backbone (outputs hidden_states) and — with lm_head kept —
+                           the ASR decoder. One prefill(embeds)->hidden + step(embeds)->hidden loop.
+  * OnnxOp               — thin wrapper over a single-file ONNX session (encoders/decoder/
+                           connector/diffusion_head/projector).
+  * DiffusionSampler     — the DDPM/DPM denoise loop (diffusion_head.onnx + codes/ scheduler + CFG).
+  * audio load / normalize (-25 dBFS) / save  — 24 kHz mono (VibeVoice standard).
+  * load_tokenizer, load_scaling — Qwen2.5 tokenizer + the stored speech scaling/bias factors.
+
+These wrap the ALREADY-verified ONNX sub-parts (see eval.py); they don't re-load PyTorch.
+Bit-exact TTS text prompts require the original VibeVoice processor/tokenizer assets (the repos
+ship none — we fetch the plain Qwen2.5 tokenizer); where that matters it's called out in the driver.
+"""
+import glob
+import json
+import sys
+import types
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).parent
+sys.path.insert(0, str(HERE))
+for _s in (sys.stdout, sys.stderr):
+    try: _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception: pass
+
+import optimize as O
+
+SR = 24000                     # VibeVoice sample rate
+TARGET_DB_FS = -25.0
+VAE_DIM = 64
+
+
+# --------------------------------------------------------------------------- resolve / io
+def resolve(model_arg, device="cpu", precision="int4", onnx_dir=None):
+    key, src = O.resolve_target(model_arg)
+    od = Path(onnx_dir) if onnx_dir else (src / f"{device}_{precision}" / "models")
+    return key, src, od
+
+
+# Friendly model ids for display.
+MODEL_IDS = {
+    "1.5b": "microsoft/VibeVoice-1.5B",
+    "asr": "microsoft/VibeVoice-ASR",
+    "asr-hf": "microsoft/VibeVoice-ASR-HF",
+    "realtime": "microsoft/VibeVoice-Realtime-0.5B",
+}
+
+
+def resolve_from_path(model_path):
+    """Inference input: a built ONNX dir in the new layout onnx/{key}/{device}_{precision}
+    (e.g. onnx/asr/cuda_fp16). Also accepts the legacy <checkpoint>/{device}_{precision}[/models].
+
+    Returns (key, src_checkpoint, onnx_dir, device, precision, model_id). SystemExit on bad path.
+    """
+    p = Path(model_path).expanduser().resolve()
+    if not p.exists():
+        raise SystemExit(f"path does not exist: {p} (build it: uv run optimize.py <model>)")
+    if list(p.glob("*.onnx")):
+        onnx_dir = p
+    elif (p / "models").is_dir() and list((p / "models").glob("*.onnx")):
+        onnx_dir = p / "models"
+    else:
+        raise SystemExit(f"no .onnx files under {p} — pass a built dir, "
+                         f"e.g. onnx/asr/cuda_fp16 (run: uv run optimize.py <model>)")
+    dp = onnx_dir.parent if onnx_dir.name == "models" else onnx_dir
+    device, sep, precision = dp.name.partition("_")
+    if not sep or device not in ("cpu", "cuda"):
+        raise SystemExit(f"cannot parse device/precision from '{dp.name}' — expected "
+                         f"'<device>_<precision>' (e.g. cpu_int4, cuda_fp16)")
+    if dp.parent.name in O.MODELS:                          # new layout: onnx/{key}/{dev}_{prec}
+        key = dp.parent.name
+        src = (HERE / O.MODELS[key]["dir"]).resolve()
+    elif (dp.parent / "config.json").exists():             # legacy: <checkpoint>/{dev}_{prec}[/models]
+        src = dp.parent
+        key = O.detect_model_type(src)
+    else:
+        raise SystemExit(f"cannot locate the checkpoint for {dp} — expected an onnx/<key>/... layout "
+                         f"or a config.json in {dp.parent}")
+    if not (src / "config.json").exists():
+        raise SystemExit(f"checkpoint dir missing config.json: {src}")
+    return key, src, onnx_dir, device, precision, MODEL_IDS.get(key, key)
+
+
+def _ort():
+    import onnxruntime as ort
+    return ort
+
+
+def _providers(device):
+    return ["CUDAExecutionProvider", "CPUExecutionProvider"] if device == "cuda" else ["CPUExecutionProvider"]
+
+
+class OnnxOp:
+    """Single-input/-output-ish ONNX op. run(**named_np) -> first output array."""
+    def __init__(self, path, device="cpu"):
+        self.sess = _ort().InferenceSession(str(path), providers=_providers(device))
+        self.inames = [i.name for i in self.sess.get_inputs()]
+
+    def run(self, **feed):
+        feed = {k: np.asarray(v, dtype=np.float32) if v.dtype != np.int64 else v for k, v in feed.items()}
+        return self.sess.run(None, {k: feed[k] for k in self.inames})[0]
+
+
+# --------------------------------------------------------------------------- audio
+def load_audio(path, sr=SR):
+    import soundfile as sf
+    wav, in_sr = sf.read(path, dtype="float32", always_2d=False)
+    if wav.ndim > 1:
+        wav = wav.mean(1)
+    if in_sr != sr:
+        import librosa
+        wav = librosa.resample(wav, orig_sr=in_sr, target_sr=sr)
+    return wav.astype(np.float32)
+
+
+def normalize_audio(wav, target_db_fs=TARGET_DB_FS, eps=1e-6):
+    rms = np.sqrt(np.mean(wav ** 2)) + eps
+    wav = wav * (10 ** (target_db_fs / 20) / rms)
+    peak = np.abs(wav).max()
+    if peak > 0.99:
+        wav = wav * (0.99 / peak)
+    return wav.astype(np.float32)
+
+
+def save_wav(path, wav, sr=SR):
+    import soundfile as sf
+    wav = np.asarray(wav, dtype=np.float32).ravel()
+    sf.write(str(path), wav, sr)
+    return path
+
+
+# --------------------------------------------------------------------------- tokenizer / scaling
+def load_tokenizer(onnx_dir, src):
+    """Qwen2.5 tokenizer saved next to the ONNX (by ModelBuilder) or the standalone dir."""
+    from transformers import AutoTokenizer
+    for cand in (onnx_dir, onnx_dir.parent.parent, src):
+        if (Path(cand) / "tokenizer.json").exists() or (Path(cand) / "tokenizer_config.json").exists():
+            return AutoTokenizer.from_pretrained(str(cand))
+    return AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
+
+
+def load_scaling(src):
+    """model.speech_scaling_factor / model.speech_bias_factor (stored scalars) or (1.0, 0.0)."""
+    from safetensors.torch import load_file
+    scale, bias = 1.0, 0.0
+    for sf in glob.glob(str(Path(src) / "*.safetensors")):
+        d = load_file(sf)
+        for k, v in d.items():
+            if k.endswith("speech_scaling_factor"):
+                scale = float(v.reshape(-1)[0])
+            elif k.endswith("speech_bias_factor"):
+                bias = float(v.reshape(-1)[0])
+    return scale, bias
+
+
+# --------------------------------------------------------------------------- codes/ scheduler
+def make_scheduler(diffusion_cfg):
+    """Instantiate codes/ DPMSolverMultistepScheduler from a diffusion_head_config dict."""
+    for name, sub in [("vibevoice", ""), ("vibevoice.schedule", "schedule")]:
+        m = types.ModuleType(name); m.__path__ = [str(HERE / "codes" / "vibevoice" / sub)]
+        sys.modules.setdefault(name, m)
+    import importlib
+    dpm = importlib.import_module("vibevoice.schedule.dpm_solver")
+    beta = diffusion_cfg.get("ddpm_beta_schedule", "cosine")
+    return dpm.DPMSolverMultistepScheduler(
+        num_train_timesteps=diffusion_cfg.get("ddpm_num_steps", 1000),
+        beta_schedule=beta,
+        prediction_type=diffusion_cfg.get("prediction_type", "v_prediction"),
+        algorithm_type="dpmsolver++",
+    )
+
+
+class DiffusionSampler:
+    """DDPM/DPM sampling of one acoustic latent per frame via diffusion_head.onnx + CFG.
+
+    condition/neg_condition: [H] float32 (LLM hidden state / negative). Returns latent [B,64]."""
+    def __init__(self, head_op: OnnxOp, diffusion_cfg, device="cpu"):
+        self.head = head_op
+        self.cfg = diffusion_cfg
+        self.steps = int(diffusion_cfg.get("ddpm_num_inference_steps", 20))
+
+    def sample(self, condition, neg_condition=None, cfg_scale=1.3, n_frames=1, seed=0):
+        import torch
+        sched = make_scheduler(self.cfg)
+        sched.set_timesteps(self.steps)
+        rng = np.random.default_rng(seed)
+        cond = np.asarray(condition, dtype=np.float32).reshape(1, -1).repeat(n_frames, 0)
+        use_cfg = neg_condition is not None and cfg_scale != 1.0
+        if use_cfg:
+            neg = np.asarray(neg_condition, dtype=np.float32).reshape(1, -1).repeat(n_frames, 0)
+            cond_all = np.concatenate([cond, neg], 0)      # [2n, H]: cond half, uncond half
+        else:
+            cond_all = cond
+        # ONE sample of n frames. CFG runs the SAME sample through both cond and uncond branches
+        # (duplicate for the head call only) — the reference does this each step; denoising must not
+        # mix guidance with a noise difference between two independent samples.
+        x = rng.standard_normal((n_frames, VAE_DIM)).astype(np.float32)
+        for t in sched.timesteps:
+            xin = np.concatenate([x, x], 0) if use_cfg else x
+            tf = np.full((xin.shape[0],), float(t), dtype=np.float32)
+            eps = self.head.run(noisy_images=xin, timesteps=tf, condition=cond_all)
+            if use_cfg:
+                c, u = np.split(eps, 2, 0)
+                eps = u + cfg_scale * (c - u)              # [n, 64]
+            out = sched.step(torch.from_numpy(eps), t, torch.from_numpy(x))
+            x = out.prev_sample.numpy().astype(np.float32)
+        return x
+
+
+# --------------------------------------------------------------------------- LLM (raw ORT KV cache)
+class OnnxLLM:
+    """Drives a genai-exported decoder (llm_decoder.onnx) directly via onnxruntime.
+
+    Inputs:  inputs_embeds[B,S,H], attention_mask[B,T], past_key_values.{i}.{key,value}[B,kv,P,hd]
+    Outputs: hidden_states[B,S,H], present.{i}.{key,value}
+    Rotary/positions are computed inside the GQA op from the attention_mask, so we only grow a
+    ones mask. Stateful: prefill(embeds) then step(embeds); .hidden holds the last hidden states."""
+    def __init__(self, path, device="cpu"):
+        import onnx
+        self.sess = _ort().InferenceSession(str(path), providers=_providers(device))
+        g = onnx.load(str(path), load_external_data=False).graph
+        self.in_names = [i.name for i in self.sess.get_inputs()]
+        kv = [n for n in self.in_names if n.startswith("past_key_values.")]
+        self.n_layers = len({n.split(".")[1] for n in kv})
+        shp = next(i for i in g.input if i.name == "past_key_values.0.key").type.tensor_type.shape.dim
+        self.kv_heads = shp[1].dim_value
+        self.head_dim = shp[3].dim_value
+        outs = [o.name for o in self.sess.get_outputs()]
+        # TTS backbone emits 'hidden_states'; an ASR decoder that kept lm_head emits 'logits'.
+        self.out_name = "logits" if "logits" in outs else "hidden_states"
+        self.hidden = None
+        self._reset()
+
+    def _reset(self, batch=1):
+        z = np.zeros((batch, self.kv_heads, 0, self.head_dim), dtype=np.float32)
+        self.past = {}
+        for i in range(self.n_layers):
+            self.past[f"past_key_values.{i}.key"] = z.copy()
+            self.past[f"past_key_values.{i}.value"] = z.copy()
+        self.total = 0
+        self.batch = batch
+
+    def _run(self, embeds):
+        embeds = np.asarray(embeds, dtype=np.float32)
+        b, s, _ = embeds.shape
+        self.total += s
+        feed = {"inputs_embeds": embeds,
+                "attention_mask": np.ones((b, self.total), dtype=np.int64)}
+        feed.update(self.past)
+        outs = self.sess.run(None, feed)
+        names = [o.name for o in self.sess.get_outputs()]
+        out = dict(zip(names, outs))
+        for i in range(self.n_layers):
+            self.past[f"past_key_values.{i}.key"] = out[f"present.{i}.key"]
+            self.past[f"past_key_values.{i}.value"] = out[f"present.{i}.value"]
+        self.hidden = out[self.out_name]
+        return self.hidden
+
+    def prefill(self, inputs_embeds):
+        """inputs_embeds [B,S,H] -> hidden [B,S,H]. Resets state to a fresh sequence."""
+        self._reset(batch=inputs_embeds.shape[0])
+        return self._run(inputs_embeds)
+
+    def step(self, inputs_embeds):
+        """inputs_embeds [B,s,H] appended to the running KV cache -> hidden [B,s,H]."""
+        return self._run(inputs_embeds)
+
+
+# The exact embed_tokens key per model — the ONNX decoder excludes embeddings, so inference looks
+# them up here. Realtime ships BOTH a base language_model AND a tts_language_model embed table (they
+# differ); the exported decoder is the tts backbone, so we must match that exact key, not first-win.
+EMBED_KEY = {
+    "1.5b": "model.language_model.embed_tokens.weight",
+    "asr": "model.language_model.embed_tokens.weight",
+    "asr-hf": "language_model.model.embed_tokens.weight",
+    "realtime": "model.tts_language_model.embed_tokens.weight",
+}
+
+
+def embed_tokens(src, token_ids, model_key):
+    """Look up token embeddings for the exported decoder of `model_key`.
+    Returns [1, len, H] float32. token_ids: 1-D list/array."""
+    from safetensors.torch import load_file
+    key = EMBED_KEY.get(model_key)
+    if key is None:
+        raise RuntimeError(f"no embed_tokens key mapping for model '{model_key}'")
+    ids = np.asarray(token_ids, dtype=np.int64).ravel()
+    for sf in glob.glob(str(Path(src) / "*.safetensors")):
+        d = load_file(sf)
+        if key in d:
+            return d[key].float().numpy()[ids][None].astype(np.float32)
+    raise RuntimeError(f"{key} not found in {src}")
+
+
+def acoustic_decode_to_wav(dec_op: OnnxOp, latents, scale, bias):
+    """latents [B,T,64] (LLM-space) -> waveform. Applies /scale - bias then acoustic_decoder."""
+    lat = (np.asarray(latents, dtype=np.float32) / (scale if scale else 1.0)) - bias
+    return dec_op.run(latents=lat)
+
+
+def component_path(onnx_dir, comp):
+    p = Path(onnx_dir) / f"{comp}.onnx"
+    return p if p.exists() else None

cpu_fp16/acoustic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cf54e4869bbfe70ea74b784f3d67597b0691b3eac5f23b9213e86cdb5a1e4b6
+size 4936693

cpu_fp16/acoustic_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9b59b84ea79d1de3d8c6a477822377437bbba2fe096d74ee913233482bf1fef
+size 688798049

cpu_fp16/acoustic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df85a2dbd67230632bfe2c517c55f5c481bab7cca8b89164f1411885414f2f7e
+size 688593181

cpu_fp16/chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

cpu_fp16/diffusion_head.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97659f98af3bbcbeac37a544b08d8baaf4c7a431f0b2d47a595ae31e861ca2d8
+size 246711153

cpu_fp16/genai_config.json

@@ -0,0 +1,49 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 65536,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "llm_decoder.onnx",
+            "head_size": 128,
+            "hidden_size": 1536,
+            "inputs": {
+                "inputs_embeds": "inputs_embeds",
+                "attention_mask": "attention_mask",
+                "position_ids": "position_ids",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 12,
+            "num_hidden_layers": 28,
+            "num_key_value_heads": 2
+        },
+        "eos_token_id": null,
+        "pad_token_id": null,
+        "type": "qwen2",
+        "vocab_size": 151936
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 65536,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": false,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

cpu_fp16/llm_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71f5c2d4e8ab8c4bb0011d1fa26c715774d3d29f9c2a7e45c2c9f8ccbfdc61ba
+size 693048

cpu_fp16/llm_decoder.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d72b898d2f9c5d623df4d932ca989eff9f6c90ef8dca33f1374fabc719be891f
+size 2637496320

cpu_fp16/semantic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:829d24a680c7f68c496a2d0274965303e71a13351c0527de099e5466e06e7717
+size 5133304

cpu_fp16/semantic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0aad94f6d44dd4f20a570a5b2c17c9b2d4668f2dacb59a70e2c88e3c01ac0f91
+size 690443001

cpu_fp16/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892

cpu_fp16/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 65536,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

cpu_fp32/acoustic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf7183835daf1959ab105f4886b3a9867a0b65fe95c93ea832b46c0dd4e3d4d2
+size 9861072

cpu_fp32/acoustic_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15625fabb679e4bf01138118e7349e055e2e3dfd8b4546b1c108c09ad0ca160e
+size 1376190019

cpu_fp32/acoustic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6d5a0f7cfd27e06e39a6f11bcbc1abba63245e9e6b256b407e2d43bd67396ad
+size 1375985271

cpu_fp32/chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

cpu_fp32/diffusion_head.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:063928366830acd04eb21b15fb3f05a08a064d0c077e49dd30a2bfe2901f505e
+size 493270097

cpu_fp32/genai_config.json

@@ -0,0 +1,48 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 65536,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "llm_decoder.onnx",
+            "head_size": 128,
+            "hidden_size": 1536,
+            "inputs": {
+                "inputs_embeds": "inputs_embeds",
+                "attention_mask": "attention_mask",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 12,
+            "num_hidden_layers": 28,
+            "num_key_value_heads": 2
+        },
+        "eos_token_id": null,
+        "pad_token_id": null,
+        "type": "qwen2",
+        "vocab_size": 151936
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 65536,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": true,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

cpu_fp32/llm_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:710b47c678f45ae232c1d8ee694642f90852ce03286ded62bee3d9ed012b9006
+size 170303

cpu_fp32/llm_decoder.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8d42ad79c9791300404c674ff6d9964910b7337e9a749fe3e6eff193a9b9efc
+size 5274927104

cpu_fp32/semantic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1403fdabeef9ca7d4fcf8177bd071f931ff8135ad64f2b0d186cdcbdb8b1c215
+size 10254291

cpu_fp32/semantic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77dd17bc1f5a3cb5d21b6c337e4d5e1a883f2a6d79373c3155dee215ba6fd591
+size 1379670225

cpu_fp32/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892

cpu_fp32/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 65536,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

cpu_int4/acoustic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf7183835daf1959ab105f4886b3a9867a0b65fe95c93ea832b46c0dd4e3d4d2
+size 9861072

cpu_int4/acoustic_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15625fabb679e4bf01138118e7349e055e2e3dfd8b4546b1c108c09ad0ca160e
+size 1376190019

cpu_int4/acoustic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6d5a0f7cfd27e06e39a6f11bcbc1abba63245e9e6b256b407e2d43bd67396ad
+size 1375985271

cpu_int4/chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

cpu_int4/diffusion_head.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:063928366830acd04eb21b15fb3f05a08a064d0c077e49dd30a2bfe2901f505e
+size 493270097

cpu_int4/genai_config.json

@@ -0,0 +1,48 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 65536,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "llm_decoder.onnx",
+            "head_size": 128,
+            "hidden_size": 1536,
+            "inputs": {
+                "inputs_embeds": "inputs_embeds",
+                "attention_mask": "attention_mask",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 12,
+            "num_hidden_layers": 28,
+            "num_key_value_heads": 2
+        },
+        "eos_token_id": null,
+        "pad_token_id": null,
+        "type": "qwen2",
+        "vocab_size": 151936
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 65536,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": true,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

cpu_int4/llm_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb58c302f5ca2927101b0e080d973b0d8740a4d3ac11a61a7052c597b616df2f
+size 222061

cpu_int4/llm_decoder.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6be0d6d7da8a77a6fe37810483c09897ce25d91038854d407545112325a98ea7
+size 857145344

cpu_int4/semantic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1403fdabeef9ca7d4fcf8177bd071f931ff8135ad64f2b0d186cdcbdb8b1c215
+size 10254291

cpu_int4/semantic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77dd17bc1f5a3cb5d21b6c337e4d5e1a883f2a6d79373c3155dee215ba6fd591
+size 1379670225

cpu_int4/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892

cpu_int4/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 65536,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

eval.py

@@ -0,0 +1,243 @@
+"""Evaluate exported VibeVoice ONNX sub-parts against their PyTorch references.
+
+Two stages:
+  A) COMPONENT parity — for every exported component (acoustic/semantic encoders & decoder,
+     diffusion_head, connectors, projector) load the PyTorch reference (user_script loaders) and
+     the ONNX session on the SAME inputs, then report cosine + max|Δ| with a PASS/FAIL threshold.
+     The LLM decoder is checked structurally (int4 MatMulNBits + GQA + inputs_embeds→hidden).
+  B) WHOLE pipeline — an end-to-end integrity chain of the ONNX pieces:
+       • TTS codec round-trip (audio → acoustic_encoder → acoustic_decoder → audio′): waveform
+         correlation + SNR — the standard neural-codec reconstruction metric.
+       • Full ONNX chain smoke: latent → acoustic_connector → (cond) → diffusion_head → 1 denoise
+         step, and acoustic_decoder → waveform — every exported block run in sequence, shapes +
+         finiteness verified. (The autoregressive LLM + multi-step DDPM loop need the genai
+         runtime and are out of scope here — reported as SKIPPED, not silently passed.)
+
+Model is the FINAL positional (a key {1.5b,asr,asr-hf,realtime} or a checkpoint dir path).
+
+Usage:
+  uv run eval.py 1.5b
+  uv run eval.py --device cuda --precision fp16 ./asr-hf
+  uv run eval.py --audio sample.wav 1.5b
+  uv run eval.py --components acoustic_encoder diffusion_head realtime
+"""
+import argparse
+import sys
+import os
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).parent
+sys.path.insert(0, str(HERE))
+for _s in (sys.stdout, sys.stderr):
+    try: _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception: pass
+
+import optimize as O          # registry (MODELS/DEVICES), resolve_target, detect
+import user_script as us
+
+COS_TOL = 0.99                # component parity threshold
+
+
+def _np(o):
+    """Best-effort tensor → float32 numpy from a torch tensor or an output object."""
+    import torch
+    if torch.is_tensor(o):
+        return o.detach().float().cpu().numpy()
+    for a in ("latents", "mean", "sample", "audio", "last_hidden_state"):
+        if hasattr(o, a):
+            v = getattr(o, a); v = v() if callable(v) else v
+            if torch.is_tensor(v):
+                return v.detach().float().cpu().numpy()
+    if isinstance(o, (tuple, list)):
+        return _np(o[0])
+    raise TypeError(f"cannot convert output of type {type(o).__name__}")
+
+
+def _cos(a, b):
+    a, b = a.ravel().astype(np.float64), b.ravel().astype(np.float64)
+    n = np.linalg.norm(a) * np.linalg.norm(b)
+    return float(a @ b / (n + 1e-9))
+
+
+def _session(path):
+    import onnxruntime as ort
+    return ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])
+
+
+def parity_component(model_key, src, comp, onnx_path):
+    """Run PyTorch ref + ONNX on identical dummy inputs; return (cos, maxd, ok)."""
+    import torch
+    loader_name, io_name, dummy_name, env = O.MODELS[model_key]["olive"][comp]
+    os.environ.update(env)
+    pyt = getattr(us, loader_name)(str(src))
+    if hasattr(pyt, "eval"):
+        pyt = pyt.eval()
+    io = getattr(us, io_name)()
+    names = io["input_names"]
+    inputs = getattr(us, dummy_name)()
+    with torch.no_grad():
+        ref = _np(pyt(*[inputs[n] for n in names]))
+    got = _session(onnx_path).run(None, {n: inputs[n].numpy() for n in names})[0]
+    m = min(ref.size, got.size)
+    a, b = got.ravel()[:m].astype(np.float64), ref.ravel()[:m].astype(np.float64)
+    cos = _cos(a, b)
+    maxd = float(np.abs(a - b).max())
+    rel = maxd / (float(np.abs(b).max()) + 1e-12)
+    # PASS on cosine, OR when tensors are numerically identical (near-zero-signal inputs make
+    # cosine noise-dominated even though maxd≈0 — e.g. a decoder fed random latents → ~silence).
+    ok = (cos >= COS_TOL) or (maxd < 1e-4) or (rel < 1e-3)
+    return cos, maxd, ok
+
+
+def check_llm(onnx_path):
+    """Structural check of the genai-built decoder (no full runtime inference)."""
+    import onnx
+    g = onnx.load(str(onnx_path), load_external_data=False).graph
+    ops = [n.op_type for n in g.node]
+    innames = [i.name for i in g.input]
+    nb, gqa = ops.count("MatMulNBits"), ops.count("GroupQueryAttention")
+    embeds_in = any("inputs_embeds" in n for n in innames)
+    ok = (nb > 0 or "MatMul" in ops) and embeds_in
+    return f"MatMulNBits={nb} GQA={gqa} inputs_embeds={embeds_in}", ok
+
+
+# ---------------------------------------------------------------- whole pipeline
+
+def _load_audio(path, samples, sr=24000):
+    import soundfile as sf, librosa
+    wav, in_sr = sf.read(path, dtype="float32", always_2d=False)
+    if wav.ndim > 1:
+        wav = wav.mean(1)
+    if in_sr != sr:
+        wav = librosa.resample(wav, orig_sr=in_sr, target_sr=sr)
+    if len(wav) < samples:
+        wav = np.pad(wav, (0, samples - len(wav)))
+    return wav[:samples].astype(np.float32)[None, None, :]
+
+
+def _synth_audio(samples, sr=24000):
+    t = np.arange(samples) / sr
+    wav = 0.3 * (np.sin(2 * np.pi * 220 * t) + 0.5 * np.sin(2 * np.pi * 440 * t))
+    return wav.astype(np.float32)[None, None, :]
+
+
+def whole_pipeline(model_key, models_dir, audio_path):
+    """End-to-end ONNX integrity: codec round-trip + full-chain smoke. Returns list of rows."""
+    rows = []
+    present = {c: models_dir / f"{c}.onnx" for c in
+              ("acoustic_encoder", "acoustic_decoder", "acoustic_connector", "diffusion_head",
+               "semantic_encoder", "multi_modal_projector")
+              if (models_dir / f"{c}.onnx").exists()}
+
+    # --- codec round-trip (encoder + decoder both present) ---
+    if "acoustic_encoder" in present and "acoustic_decoder" in present:
+        enc, dec = _session(present["acoustic_encoder"]), _session(present["acoustic_decoder"])
+        samples = enc.get_inputs()[0].shape[2]
+        samples = 24000 if not isinstance(samples, int) else samples
+        wav = _load_audio(audio_path, samples) if audio_path else _synth_audio(samples)
+        lat = enc.run(None, {"audio": wav})[0]
+        dec_frames = dec.get_inputs()[0].shape[1]
+        if isinstance(dec_frames, int) and lat.shape[1] != dec_frames:
+            f = min(lat.shape[1], dec_frames)
+            lat = lat[:, :f, :] if lat.shape[1] > f else np.pad(lat, ((0, 0), (0, dec_frames - lat.shape[1]), (0, 0)))
+        recon = dec.run(None, {"latents": lat.astype(np.float32)})[0].ravel()
+        n = min(len(recon), wav.size)
+        x = wav.ravel()[:n]; y = recon[:n]
+        corr = float(np.corrcoef(x, y)[0, 1]) if n > 1 else float("nan")
+        noise = x - y
+        snr = 10 * np.log10((np.sum(x**2) + 1e-12) / (np.sum(noise**2) + 1e-12))
+        finite = bool(np.isfinite(recon).all())
+        rows.append(("codec round-trip", f"corr={corr:+.3f} SNR={snr:+.1f}dB finite={finite} "
+                     f"in={wav.size} out={len(recon)}", finite))
+    else:
+        rows.append(("codec round-trip", "SKIPPED (needs acoustic_encoder+decoder)", None))
+
+    # --- full ONNX chain smoke: latent → connector → diffusion_head; latent → decoder ---
+    if "acoustic_connector" in present and "diffusion_head" in present:
+        conn = _session(present["acoustic_connector"]); dh = _session(present["diffusion_head"])
+        rng = np.random.default_rng(0)
+        T = 8
+        lat = rng.standard_normal((1, T, 64)).astype(np.float32)
+        cond = conn.run(None, {"features": lat})[0]                 # [1,T,H]
+        H = cond.shape[-1]
+        ni = rng.standard_normal((T, 64)).astype(np.float32)
+        ts = (rng.random(T) * 1000).astype(np.float32)
+        pred = dh.run(None, {"noisy_images": ni, "timesteps": ts,
+                             "condition": cond.reshape(T, H).astype(np.float32)})[0]
+        ok = pred.shape == (T, 64) and np.isfinite(pred).all() and np.isfinite(cond).all()
+        rows.append(("tts chain smoke", f"connector→[H={H}]→diffusion_head pred={pred.shape} finite={ok}", ok))
+    else:
+        rows.append(("tts chain smoke", "SKIPPED (needs acoustic_connector+diffusion_head)", None))
+
+    # --- ASR fusion chain: encoders → projector/connectors → fused features ---
+    if "multi_modal_projector" in present and "acoustic_encoder" in present and "semantic_encoder" in present:
+        ae, se, proj = (_session(present[k]) for k in ("acoustic_encoder", "semantic_encoder", "multi_modal_projector"))
+        wav = _synth_audio(24000)
+        al = ae.run(None, {"audio": wav})[0]; sl = se.run(None, {"audio": wav})[0]
+        f = min(al.shape[1], sl.shape[1])
+        fused = proj.run(None, {"acoustic_latents": al[:, :f].astype(np.float32),
+                                "semantic_latents": sl[:, :f].astype(np.float32)})[0]
+        ok = np.isfinite(fused).all()
+        rows.append(("asr fusion chain", f"ac+sem→projector fused={fused.shape} finite={ok}", ok))
+
+    rows.append(("autoregressive LLM + DDPM loop", "SKIPPED (needs genai runtime; out of scope)", None))
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Evaluate VibeVoice ONNX sub-parts",
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    global COS_TOL
+    ap.add_argument("--device", choices=["cpu", "cuda", "gpu"], default="cpu")
+    ap.add_argument("--precision", choices=["int4", "fp16", "fp32"], default="int4")
+    ap.add_argument("--components", help="comma-separated subset (default: all present)")
+    ap.add_argument("--output-dir", help="models dir (default onnx/{model}/{device}_{precision})")
+    ap.add_argument("--audio", help="wav for codec round-trip (default: synthetic tone)")
+    ap.add_argument("--tol", type=float, default=COS_TOL, help=f"cosine PASS threshold (default {COS_TOL})")
+    ap.add_argument("model", help="keyword {1.5b,asr,asr-hf,realtime} or checkpoint dir path")
+    args = ap.parse_args()
+    COS_TOL = args.tol
+    device = "cuda" if args.device == "gpu" else args.device
+    model_key, src = O.resolve_target(args.model)
+    models_dir = Path(args.output_dir) if args.output_dir else (HERE / "onnx" / model_key / f"{device}_{args.precision}")
+    if not models_dir.exists():
+        sys.exit(f"no exported models at {models_dir} — run optimize.py first")
+
+    print(f"=== eval {model_key}  src={src}  models={models_dir}  tol={COS_TOL} ===\n")
+
+    # ---- A) component parity ----
+    comps = [c.strip() for c in args.components.split(",")] if args.components else O.all_components(model_key)
+    print("A) COMPONENT PARITY (ONNX vs PyTorch)")
+    results = []
+    for comp in comps:
+        onnx_path = models_dir / (f"{comp}.onnx" if comp != "llm" else "llm_decoder.onnx")
+        if not onnx_path.exists():
+            print(f"  {comp:22s}  — not exported (skip)"); continue
+        try:
+            if comp == "llm":
+                info, ok = check_llm(onnx_path)
+                print(f"  {comp:22s}  {'PASS' if ok else 'FAIL'}  {info}"); results.append(ok)
+            else:
+                cos, maxd, ok = parity_component(model_key, src, comp, onnx_path)
+                print(f"  {comp:22s}  {'PASS' if ok else 'FAIL'}  cos={cos:.6f} maxd={maxd:.2e}")
+                results.append(ok)
+        except Exception as e:
+            print(f"  {comp:22s}  ERROR  {type(e).__name__}: {str(e)[:120]}"); results.append(False)
+
+    # ---- B) whole pipeline ----
+    print("\nB) WHOLE PIPELINE (end-to-end ONNX integrity)")
+    for name, info, ok in whole_pipeline(model_key, models_dir, args.audio):
+        tag = "SKIP" if ok is None else ("PASS" if ok else "FAIL")
+        print(f"  {name:32s}  {tag}  {info}")
+        if ok is not None:
+            results.append(ok)
+
+    npass = sum(1 for r in results if r); ntot = len(results)
+    print(f"\n=== {npass}/{ntot} checks passed ===")
+    sys.exit(0 if npass == ntot else 1)
+
+
+if __name__ == "__main__":
+    main()

gpu_fp16/acoustic_connector.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cf54e4869bbfe70ea74b784f3d67597b0691b3eac5f23b9213e86cdb5a1e4b6
+size 4936693

gpu_fp16/acoustic_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9b59b84ea79d1de3d8c6a477822377437bbba2fe096d74ee913233482bf1fef
+size 688798049

gpu_fp16/acoustic_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df85a2dbd67230632bfe2c517c55f5c481bab7cca8b89164f1411885414f2f7e
+size 688593181

gpu_fp16/chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

gpu_fp16/diffusion_head.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97659f98af3bbcbeac37a544b08d8baaf4c7a431f0b2d47a595ae31e861ca2d8
+size 246711153

gpu_fp16/genai_config.json

@@ -0,0 +1,55 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 65536,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": [
+                    {
+                        "cuda": {
+                            "enable_cuda_graph": "0",
+                            "enable_skip_layer_norm_strict_mode": "1"
+                        }
+                    }
+                ]
+            },
+            "filename": "llm_decoder.onnx",
+            "head_size": 128,
+            "hidden_size": 1536,
+            "inputs": {
+                "inputs_embeds": "inputs_embeds",
+                "attention_mask": "attention_mask",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 12,
+            "num_hidden_layers": 28,
+            "num_key_value_heads": 2
+        },
+        "eos_token_id": null,
+        "pad_token_id": null,
+        "type": "qwen2",
+        "vocab_size": 151936
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 65536,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": true,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

gpu_fp16/llm_decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59e36731e6eb9593445c2d8e925e88290a364586c675bf2cbd7efd6fb516f1d8
+size 170213

gpu_fp16/llm_decoder.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b1adc93736ca403011ad3cb64614d8185f347142f5b404f6604aebe14eec0c8
+size 2637496320