MiniMax-H3 Wushu Action LoRA demo

README.md

@@ -1,13 +1,70 @@
 ---
-title: Wushu Action H3 Lora Demo
-emoji: 🐢
+title: MiniMax-H3 Wushu Action LoRA
+emoji: 🥋
 colorFrom: blue
-colorTo: blue
+colorTo: green
 sdk: gradio
 sdk_version: 6.24.0
-python_version: '3.12'
 app_file: app.py
 pinned: false
+short_description: Martial-arts action video with a MiniMax-H3 LoRA
+python_version: "3.12"
+startup_duration_timeout: 30m
+suggested_hardware: zero-a10g
+models:
+  - MiniMaxAI/MiniMax-H3
+  - Jojocodex/minimax-h3-wushu-action-lora
+  - Comfy-Org/MiniMax-H3
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 武打动作 · MiniMax-H3 Wushu Action LoRA
+
+Generate short **martial-arts / kung-fu action** clips — punches, kicks, combination forms, staff technique — with
+[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) and the
+[Wushu Action LoRA](https://huggingface.co/Jojocodex/minimax-h3-wushu-action-lora). Every clip comes back with
+H3's native synchronized stereo soundtrack.
+
+## What it does
+
+The LoRA was trained with ai-toolkit on **455 curated wushu clips** (2000 steps, rank 16, resolution 512,
+90 frames @ 24 fps) to teach MiniMax-H3 human martial-arts motion — the four technique families its captions
+cover are 拳法 (punches), 腿法 (kicks), 综合套路 (combination forms) and 棍法 (staff).
+
+It has **no trigger token**: the model card activates it with a natural-language action description
+(*"a kung fu practitioner executing a spinning kick"*). The demo's **technique family** picker appends the
+caption wording of the family you choose to your prompt, so a request lands inside the distribution the LoRA
+was trained on. The default duration is **3.75 s = 90 frames**, exactly the clip length it saw during training.
+
+## How it works
+
+This Space is the **denoising half** of a split MiniMax-H3 deployment:
+
+- The 62 GiB Qwen3-VL text encoder runs in the
+  [conditioner Space](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), called over the gradio API;
+  `prompt_embeds` + `text_token_tags` is the whole wire format between the halves.
+- This Space loads the 61.7 GiB transformer + 10.4 GiB VAEs and runs the denoising loop and the video/audio decode
+  on the GPU (ZeroGPU, `xlarge` booking — the transformer does not fit the 48 GB default).
+- The wushu action LoRA (`wushu_action_h3_lora_v4_2000_pruned.safetensors`, 416 keys over 208 modules) is **folded**
+  into the transformer's bf16 weights at startup, at the top of the card's recommended 0.8–1.0 strength range.
+- The [Comfy-Org Turbo LoRA](https://huggingface.co/Comfy-Org/MiniMax-H3) (4-step) is folded on top — the LoRA card
+  states the pruned file (with `adaln_proj` removed) is Turbo-compatible.
+
+## Usage
+
+Describe the action — *"a fighter executing a spinning staff technique"*, *"a practitioner demonstrating a fast
+flurry of punches in continuous motion"* — optionally pick a technique family, and click **Generate**.
+
+## LoRA details
+
+| Field | Value |
+|---|---|
+| Base model | MiniMax-H3 (FL2VA) |
+| Rank | 16 |
+| Training | ai-toolkit, 2000 steps, 455 wushu clips |
+| Resolution | 512 |
+| Clip length | 90 frames @ 24 fps (3.75 s) |
+| Recommended strength | 0.8–1.0 (folded at 1.0) |
+| Turbo compatibility | Yes (pruned file, `adaln_proj` removed) |
+
+The LoRA card notes this is the **v4** checkpoint, captioned with a generic template; a per-technique captioned
+**v5** is in progress.

app.py

@@ -0,0 +1,406 @@
+"""MiniMax-H3 Wushu Action LoRA — text-to-video demo for the martial-arts / kung-fu motion LoRA.
+
+This Space is the denoising half of a split MiniMax-H3 deployment:
+- The 62 GiB Qwen3-VL text encoder runs in the conditioner Space (`multimodalart/qwen3vl-conditioner`),
+  called over the gradio API for each request.
+- This Space loads the 61.7 GiB transformer + 10.4 GiB VAEs (77.3 GB total) and runs the denoising loop
+  and the video + audio decode on the GPU.
+- The Jojocodex wushu action LoRA (rank 16, `_pruned`) is folded into the transformer weights at startup,
+  adding human martial-arts motion: punches, kicks, combination forms, staff technique.
+- The Comfy-Org MiniMax-H3 Turbo LoRA (4-step) is folded on top, which the LoRA card says is supported
+  because the `_pruned` file has its `adaln_proj` rows removed.
+
+The LoRA was trained with ai-toolkit on 455 curated wushu clips at 90 frames / 24 fps — so the demo's default
+duration is 3.75 s, which is exactly the 90-frame window it saw.
+"""
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+import traceback
+from functools import cache
+
+# Before anything that could initialize CUDA: `import spaces` patches `torch.cuda` so the 72 GiB load can happen
+# at startup rather than on GPU time.
+import spaces
+import gradio as gr
+
+# --- Configuration ---
+MODEL_REPO = os.environ.get("H3_MODEL_REPO", "MiniMaxAI/MiniMax-H3")
+CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
+LORA_REPO = "Jojocodex/minimax-h3-wushu-action-lora"
+
+# `pack` places the transformer at startup, `lazy` moves everything on the first GPU call.
+PLACEMENT = os.environ.get("H3_PLACEMENT", "pack").lower()
+# cuDNN's fused attention is 10-20% faster than the SDPA default on this pool.
+ATTENTION = os.environ.get("H3_ATTENTION", "_native_cudnn").lower()
+# The transformer alone is 61.7 GiB, so the 48 GB `large` booking cannot hold it.
+GPU_SIZE = os.environ.get("H3_GPU_SIZE", "xlarge")
+
+# --- Canvas definitions (labels are the wire contract with the conditioner) ---
+CANVASES = {
+    # 16:9
+    "960x544 · 16:9 fast": (544, 960),
+    "1024x576 · 16:9 fast": (576, 1024),
+    "1152x640 · 16:9": (640, 1152),
+    "1280x704 · 16:9": (704, 1280),
+    "1344x768 · 16:9 full": (768, 1344),
+    # 9:16
+    "544x960 · 9:16 fast": (960, 544),
+    "640x1152 · 9:16": (1152, 640),
+    "768x1344 · 9:16 full": (1344, 768),
+    # 1:1
+    "544x544 · 1:1 fast": (544, 544),
+    "768x768 · 1:1 full": (768, 768),
+    # 4:3 / 3:4
+    "768x576 · 4:3 fast": (576, 768),
+    "1024x768 · 4:3 full": (768, 1024),
+    "576x768 · 3:4 fast": (768, 576),
+    "768x1024 · 3:4 full": (1024, 768),
+    # 21:9
+    "1152x512 · 21:9 fast": (512, 1152),
+    "1536x672 · 21:9 full": (672, 1536),
+}
+DEFAULT_CANVAS = "960x544 · 16:9 fast"
+
+FPS = 24
+FRAMES_PER_CHUNK = 17
+LATENTS_PER_CHUNK = 5
+
+MIN_UI_DURATION = 2.0
+MAX_UI_DURATION = 14.0
+# 3.75 s == 90 frames == 17 * 5 + 5, the clip length the LoRA was trained on.
+TRAINED_DURATION = 3.75
+
+DEFAULT_STEPS = 4
+DEFAULT_SEED = 42
+
+# The LoRA has no literal trigger token: the card says to activate it with a natural-language *action* description,
+# and lists the four technique families it was captioned around. Each cue below is appended to the user's prompt so
+# a request lands inside the family the user picked, in the wording the training captions used.
+TECHNIQUES: dict[str, str] = {
+    "Free-form (no cue)": "",
+    "拳法 · Punches": "fist and punch techniques, fast hand strikes in continuous motion",
+    "腿法 · Kicks": "kicking techniques, high leg strikes and spinning kicks",
+    "综合套路 · Combination forms": "a continuous martial arts form, combination techniques flowing one into the next",
+    "棍法 · Staff": "staff technique, spinning and striking with a long staff",
+}
+DEFAULT_TECHNIQUE = "Free-form (no cue)"
+
+
+def snap_frames(seconds: float) -> int:
+    """The frame count MiniMax-H3's video VAE can decode: the next `17 * n + 5` at 24 fps."""
+    frames = max(1, round(float(seconds) * FPS))
+    while frames % FRAMES_PER_CHUNK != LATENTS_PER_CHUNK:
+        frames += 1
+    return frames
+
+
+def compose_prompt(prompt: str, technique: str = DEFAULT_TECHNIQUE) -> str:
+    """The prompt the model is actually conditioned on: the request plus the technique-family cue."""
+    prompt = (prompt or "").strip()
+    cue = TECHNIQUES.get(technique or DEFAULT_TECHNIQUE, "")
+    if not cue:
+        return prompt
+    return f"{prompt.rstrip('.,;')}, {cue}"
+
+
+def lower_duration_floor(seconds: float = MIN_UI_DURATION) -> None:
+    """Let the pipeline generate below its 5 s floor — the LoRA's own clips are 3.75 s."""
+    from diffusers.modular_pipelines.minimax_h3.modular_pipeline import MiniMaxH3ModularPipeline
+
+    MiniMaxH3ModularPipeline.min_duration = property(lambda self: float(seconds))
+
+
+# --- Global state ---
+PIPE = None
+MANAGER = None
+LOAD_ERROR: str | None = None
+LOADED_IN: float | None = None
+LORA_STATUS: str | None = None
+
+
+def status() -> str:
+    if LOAD_ERROR:
+        return LOAD_ERROR
+    if PIPE is None:
+        return f"Loading `{MODEL_REPO}` (transformer + VAEs, 77.3 GB). Watch the Space logs."
+    return (
+        f"Ready · transformer + VAEs **bfloat16** · "
+        f"placement `{PLACEMENT}` · "
+        f"attention `{ATTENTION}` · "
+        f"{LORA_STATUS or 'no LoRA'} · "
+        f"loaded in {LOADED_IN:.0f}s · "
+        f"conditioner `{CONDITIONER_SPACE}`"
+    )
+
+
+def load_models() -> str | None:
+    """Load the denoising half at startup."""
+    global PIPE, MANAGER, LOAD_ERROR, LOADED_IN, LORA_STATUS
+
+    if PIPE is not None or LOAD_ERROR is not None:
+        return LOAD_ERROR
+
+    started = time.time()
+    try:
+        import torch
+        from diffusers import ComponentsManager
+
+        from h3_split_blocks import MiniMaxH3GeneratorBlocks
+
+        lower_duration_floor()
+        manager = ComponentsManager()
+        blocks = MiniMaxH3GeneratorBlocks()
+
+        print(f"[gen] loading {[c.name for c in blocks.expected_components]} from {MODEL_REPO} ...", flush=True)
+        pipe = blocks.init_pipeline(MODEL_REPO, components_manager=manager, collection="h3")
+        pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
+
+        # Fold the wushu action LoRA (+ the Turbo LoRA) into the bf16 weights.
+        import h3_lora
+
+        LORA_STATUS = h3_lora.apply_lora(pipe.transformer)
+        if LORA_STATUS:
+            print(f"[gen] {LORA_STATUS}", flush=True)
+
+        pipe.transformer.set_attention_backend(ATTENTION)
+
+        if PLACEMENT == "pack":
+            pipe.transformer.to("cuda")
+
+        PIPE = pipe
+        MANAGER = manager
+        LOADED_IN = time.time() - started
+        print(f"[gen] ready in {LOADED_IN:.0f}s", flush=True)
+    except Exception as error:
+        traceback.print_exc()
+        LOAD_ERROR = (
+            f"**Loading `{MODEL_REPO}` failed** after {time.time() - started:.0f}s: "
+            f"`{type(error).__name__}: {error}`"
+        )
+    return LOAD_ERROR
+
+
+@cache
+def conditioner():
+    """The other half, over the gradio API."""
+    from gradio_client import Client
+
+    return Client(CONDITIONER_SPACE)
+
+
+def encode_remote(prompt, canvas, num_frames):
+    """`/encode` on the conditioner Space — text only, this LoRA is text-to-video."""
+    from safetensors import safe_open
+
+    path, plan = conditioner().predict(
+        prompt=prompt,
+        image_path=None,
+        last_image_path=None,
+        canvas=canvas,
+        num_frames=num_frames,
+        rewrite_prompt=False,
+        api_name="/encode",
+    )
+    with safe_open(path, framework="pt") as handle:
+        return handle.get_tensor("prompt_embeds"), handle.get_tensor("text_token_tags"), handle.metadata(), plan
+
+
+# --- GPU duration estimation (fit on this pool: denoise is quadratic in the packed sequence length) ---
+_DUR_B = 1.1745e-4
+_DUR_C = 3.8396e-9
+_DECODE_BASE = 15
+_DECODE_PER_DEFAULT_CANVAS = 15
+_DEFAULT_CANVAS_PIXELS = 960 * 544 * 124
+_PLACEMENT_ALLOWANCE = 12
+_PAD = 10
+
+
+def get_duration(prompt_embeds, text_token_tags, height, width, num_frames, steps, seed, *a, **k):
+    height, width, num_frames, steps = int(height), int(width), int(num_frames), int(steps)
+
+    latent_frames = (num_frames - LATENTS_PER_CHUNK) // FRAMES_PER_CHUNK * LATENTS_PER_CHUNK + 2
+    rows = latent_frames * (height // 32) * (width // 32)
+    denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
+    decode = _DECODE_BASE + _DECODE_PER_DEFAULT_CANVAS * (height * width * num_frames) / _DEFAULT_CANVAS_PIXELS
+    return max(60, int(denoise + decode) + _PLACEMENT_ALLOWANCE + _PAD)
+
+
+@spaces.GPU(duration=get_duration, size=GPU_SIZE)
+def _generate(prompt_embeds, text_token_tags, height, width, num_frames, steps, seed):
+    """The only thing on GPU time: the packed-sequence denoise loop and the two decoders."""
+    import torch
+
+    if PLACEMENT == "lazy":
+        PIPE.to("cuda")
+    elif PLACEMENT == "pack":
+        PIPE.vae.to("cuda")
+        PIPE.audio_vae.to("cuda")
+
+    state = PIPE(
+        prompt_embeds=prompt_embeds.to("cuda"),
+        text_token_tags=text_token_tags,
+        image=None,
+        last_image=None,
+        height=int(height),
+        width=int(width),
+        num_frames=int(num_frames),
+        num_inference_steps=int(steps),
+        generator=torch.Generator("cpu").manual_seed(int(seed)),
+    )
+    return state.get("videos")[0], state.get("audio")[0].cpu(), state.get("sampling_rate")
+
+
+def generate(
+    prompt,
+    technique=DEFAULT_TECHNIQUE,
+    canvas=DEFAULT_CANVAS,
+    duration=TRAINED_DURATION,
+    steps=DEFAULT_STEPS,
+    seed=DEFAULT_SEED,
+    progress=gr.Progress(track_tqdm=True),
+):
+    """Generate a martial-arts action clip with the MiniMax-H3 wushu action LoRA.
+
+    Parameters:
+        prompt: What the fighter does, e.g. "a kung fu practitioner executing a spinning kick"
+        technique: Technique family cue appended to the prompt (punches / kicks / forms / staff)
+        canvas: Output resolution and aspect ratio
+        duration: Clip length in seconds, snapped to the 17n+5 frames the video VAE decodes
+        steps: Denoising steps (4 with the Turbo LoRA folded in)
+        seed: Random seed for reproducibility
+    """
+    if LOAD_ERROR:
+        raise gr.Error(LOAD_ERROR)
+    if PIPE is None:
+        raise gr.Error("The denoiser is still loading. Please wait a moment and try again.")
+    if not prompt or not prompt.strip():
+        raise gr.Error("Please describe the martial-arts action you want, e.g. 'a fighter throws a spinning kick'.")
+
+    from diffusers.utils import encode_video
+
+    full_prompt = compose_prompt(prompt, technique)
+    num_frames = snap_frames(duration)
+
+    progress(0.0, desc=f"Conditioning on {CONDITIONER_SPACE} ...")
+    conditioned = time.time()
+    prompt_embeds, text_token_tags, metadata, plan = encode_remote(full_prompt, canvas, num_frames)
+    condition_seconds = time.time() - conditioned
+    height, width, num_frames = (int(metadata[key]) for key in ("height", "width", "num_frames"))
+
+    progress(0.1, desc=f"Denoising {int(steps)} steps at {width}x{height}, {num_frames} frames ...")
+    started = time.time()
+    frames, audio, sampling_rate = _generate(
+        prompt_embeds, text_token_tags, height, width, num_frames, steps, seed
+    )
+    generate_seconds = time.time() - started
+
+    directory = os.path.join(tempfile.gettempdir(), "h3-outputs")
+    os.makedirs(directory, exist_ok=True)
+    path = os.path.join(directory, f"h3-wushu-{int(time.time() * 1000)}.mp4")
+    encode_video(frames, fps=FPS, output_path=path, audio=audio, audio_sample_rate=sampling_rate)
+
+    report = (
+        f"**Prompt sent to the model:** {full_prompt}\n\n"
+        f"`{width}x{height}`, {num_frames} frames ({num_frames / FPS:.2f} s), {int(steps)} steps · "
+        f"seed {int(seed)} · conditioner {condition_seconds:.0f}s ({plan['num_text_tokens']} tokens) · "
+        f"denoise + decode {generate_seconds:.0f}s ({generate_seconds / max(int(steps), 1):.1f} s/step)"
+    )
+    print(f"[gen] {report}", flush=True)
+    return path, report
+
+
+# --- Load models at startup ---
+load_models()
+
+INTRO = """# 武打动作 · MiniMax-H3 Wushu Action LoRA
+
+<div>
+  <a href="https://huggingface.co/Jojocodex/minimax-h3-wushu-action-lora" target="_blank" rel="noopener"><strong>[ LoRA ]</strong></a> &nbsp;
+  <a href="https://huggingface.co/MiniMaxAI/MiniMax-H3" target="_blank" rel="noopener"><strong>[ base model ]</strong></a> &nbsp;
+  <a href="https://huggingface.co/Comfy-Org/MiniMax-H3" target="_blank" rel="noopener"><strong>[ ComfyUI weights ]</strong></a>
+</div>
+
+Generate short **martial-arts / kung-fu action** clips — punches, kicks, forms, staff work — with
+[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) and the
+[Wushu Action LoRA](https://huggingface.co/Jojocodex/minimax-h3-wushu-action-lora), trained on 455 curated wushu
+clips at 90 frames / 24 fps. The video comes back with H3's native synchronized soundtrack.
+
+The LoRA has **no trigger token** — it activates on the action description itself. Describe the strike, then pick a
+technique family to steer it toward the wording its captions used. Generation takes roughly a minute.
+"""
+
+CSS = """
+.main.fillable {max-width: 1250px !important}
+.dark .gradio-container { color: var(--body-text-color); }
+"""
+
+with gr.Blocks(title="MiniMax-H3 Wushu Action LoRA") as demo:
+    gr.Markdown(INTRO)
+
+    with gr.Row(equal_height=True):
+        with gr.Column():
+            prompt = gr.Textbox(
+                label="Action",
+                lines=3,
+                placeholder="e.g. 'a kung fu practitioner executing a spinning kick on a temple courtyard'",
+                value="a martial artist performing punches and kicks in fast combat",
+            )
+            technique = gr.Radio(
+                label="Technique family",
+                info="Appended to the prompt in the wording the LoRA's captions used.",
+                choices=list(TECHNIQUES),
+                value=DEFAULT_TECHNIQUE,
+            )
+            run = gr.Button("Generate", variant="primary", size="lg")
+            with gr.Accordion("Advanced options", open=False):
+                canvas = gr.Dropdown(label="Canvas", choices=list(CANVASES), value=DEFAULT_CANVAS)
+                duration = gr.Slider(
+                    label="Duration (s)",
+                    info="Snapped up to the next 17n+5 frames. The LoRA was trained at 90 frames (3.75 s).",
+                    minimum=MIN_UI_DURATION,
+                    maximum=MAX_UI_DURATION,
+                    step=0.25,
+                    value=TRAINED_DURATION,
+                )
+                steps = gr.Slider(
+                    label="Steps",
+                    info="4 is enough with the Turbo LoRA folded in.",
+                    minimum=4,
+                    maximum=30,
+                    step=1,
+                    value=DEFAULT_STEPS,
+                )
+                seed = gr.Number(label="Seed", value=DEFAULT_SEED, precision=0)
+
+        with gr.Column():
+            video = gr.Video(label="Video + soundtrack", autoplay=True)
+            report = gr.Markdown()
+
+    gr.Examples(
+        examples=[
+            ["a martial artist performing punches and kicks in fast combat", "Free-form (no cue)"],
+            ["a kung fu practitioner executing a spinning kick", "腿法 · Kicks"],
+            ["two fighters exchanging strikes in an intense fight", "Free-form (no cue)"],
+            ["a martial artist performing a powerful roundhouse kick, with explosive force", "腿法 · Kicks"],
+            ["a practitioner demonstrating a fast flurry of punches in continuous motion", "拳法 · Punches"],
+            ["a fighter executing a spinning staff technique", "棍法 · Staff"],
+            ["a wushu athlete running through a form in a courtyard at dawn", "综合套路 · Combination forms"],
+        ],
+        inputs=[prompt, technique],
+        outputs=[video, report],
+        fn=generate,
+        cache_examples=True,
+        cache_mode="lazy",
+    )
+
+    inputs = [prompt, technique, canvas, duration, steps, seed]
+    run.click(generate, inputs, [video, report], api_name="generate")
+    prompt.submit(generate, inputs, [video, report], show_api=False)
+
+
+if __name__ == "__main__":
+    demo.launch(show_error=True, theme=gr.themes.Citrus(), css=CSS, max_threads=1000)

h3_lora.py

@@ -0,0 +1,167 @@
+"""Wushu Action LoRA support for the diffusers MiniMax-H3 transformer.
+
+The Jojocodex wushu-action LoRA (`Jojocodex/minimax-h3-wushu-action-lora`) targets the ComfyUI reference module
+tree — `diffusion_model.blocks.N.attn.qkv_proj`, `diffusion_model.blocks.N.mlp.fc1`, etc. — with rank 16 and no
+alpha metadata (so scale = 1.0, matching the convention where alpha == rank). The shipped `_pruned` file carries
+416 keys over 208 base modules (52 transformer blocks x {attn.qkv_proj, attn.out_proj, mlp.fc1, mlp.fc2}, the two
+`token_refiner` blocks included) and has its `adaln_proj` rows removed (`__metadata__: {"adaln_pruned": "true"}`),
+which is what makes it stackable with the Turbo acceleration LoRA.
+
+The LoRA is applied by folding `scale * (lora_B @ lora_A)` into the bf16 weights rather than as runtime wrappers:
+folding costs nothing per request and keeps the transformer a plain `nn.Module` for the ZeroGPU startup packing.
+Deltas are computed in float32 and round once on the way back into bf16.
+
+The diffusers conversion transforms:
+- strip the `diffusion_model.` prefix the ai-toolkit export uses
+- fused-QKV row thirds onto `attn.to_q/k/v`
+- `SwiGLU` gate/value swap onto `ff.net.0.proj`
+- `fc2` -> `ff.net.2`, `attn.out_proj` -> `attn.to_out.0`
+- `blocks.` -> `transformer_blocks.`
+- `token_refiner.blocks.` -> `token_refiner.refiner_blocks.`
+- `final_layer.adaln_proj.linear` -> `norm_out.linear` (not present in the pruned file)
+"""
+
+from __future__ import annotations
+
+import os
+
+import torch
+
+# --- Wushu Action LoRA config ---
+# The model card tells ComfyUI users to take the `_pruned` file; it is also the Turbo-compatible one.
+WUSHU_LORA_REPO = os.environ.get("WUSHU_LORA_REPO", "Jojocodex/minimax-h3-wushu-action-lora")
+WUSHU_LORA_FILE = os.environ.get("WUSHU_LORA_FILE", "wushu_action_h3_lora_v4_2000_pruned.safetensors")
+# The card recommends strength 0.8~1.0 for the ComfyUI LoraLoader; folded at the top of that range.
+WUSHU_LORA_STRENGTH = float(os.environ.get("WUSHU_LORA_STRENGTH", "1.0"))
+
+# --- Turbo LoRA config (optional acceleration) ---
+TURBO_LORA_REPO = os.environ.get("TURBO_LORA_REPO", "Comfy-Org/MiniMax-H3")
+TURBO_LORA_FILE = os.environ.get(
+    "TURBO_LORA_FILE", "loras/minimax_h3_fl2v_turbo_4step_v1.0_768p_comfyui_bf16.safetensors"
+)
+TURBO_LORA_STRENGTH = float(os.environ.get("TURBO_LORA_STRENGTH", "1.0"))
+
+USE_TURBO = os.environ.get("USE_TURBO", "1").lower() not in ("0", "off", "none", "false")
+
+
+def _targets(name: str, b: torch.Tensor, inner_dim: int) -> list[tuple[str, torch.Tensor]]:
+    """Map one reference-tree base name and its `lora_B` onto diffusers parameter key + row-transformed B."""
+    name = name.removeprefix("diffusion_model.")
+
+    if name.startswith("token_refiner.blocks."):
+        target = name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
+    elif name.startswith("blocks."):
+        target = name.replace("blocks.", "transformer_blocks.", 1)
+    else:
+        target = name
+    target = target.replace("final_layer.adaln_proj.linear", "norm_out.linear")
+
+    if target.endswith(".attn.qkv_proj"):
+        prefix = target.removesuffix("qkv_proj")
+        return [
+            (f"{prefix}to_{kind}.weight", part.contiguous())
+            for kind, part in zip(("q", "k", "v"), b.split(inner_dim, dim=0))
+        ]
+    if target.endswith(".mlp.fc1"):
+        gate, value = b.chunk(2, dim=0)
+        return [(target.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight", torch.cat([value, gate]).contiguous())]
+    if target.endswith(".mlp.fc2"):
+        return [(target.replace(".mlp.fc2", ".ff.net.2") + ".weight", b)]
+    if target.endswith(".attn.out_proj"):
+        return [(target.replace(".attn.out_proj", ".attn.to_out.0") + ".weight", b)]
+    # `adaln_proj.linear` (block-level and the final `norm_out.linear`): identical row layout on both sides.
+    return [(target + ".weight", b)]
+
+
+def _load_wushu_lora(inner_dim: int) -> dict:
+    """Load the Jojocodex wushu action LoRA from the Hub."""
+    from huggingface_hub import hf_hub_download
+    from safetensors.torch import load_file
+
+    lora = load_file(hf_hub_download(WUSHU_LORA_REPO, WUSHU_LORA_FILE))
+    bases = sorted({key.rsplit(".lora_", 1)[0] for key in lora})
+    entries = []
+    for name in bases:
+        a = lora[f"{name}.lora_A.weight"]
+        b = lora[f"{name}.lora_B.weight"]
+        entries.extend((key, a, b_part) for key, b_part in _targets(name, b, inner_dim))
+    return {
+        "label": f"{WUSHU_LORA_REPO}/{WUSHU_LORA_FILE}",
+        "scale": WUSHU_LORA_STRENGTH,  # alpha == rank, so the base scale is 1
+        "entries": entries,
+    }
+
+
+def _load_turbo_lora(inner_dim: int) -> dict:
+    """Load the MiniMax-H3 Turbo LoRA from Comfy-Org for 4-step accelerated inference.
+
+    The Comfy-Org Turbo LoRA uses the kohya format with explicit `.alpha` keys per LoRA layer, so each entry's
+    scale is `alpha / rank`, applied to `B` before the fused-QKV split.
+    """
+    from huggingface_hub import hf_hub_download
+    from safetensors.torch import load_file
+
+    lora = load_file(hf_hub_download(TURBO_LORA_REPO, TURBO_LORA_FILE))
+
+    all_keys = list(lora.keys())
+    bases = sorted({key.rsplit(".lora_", 1)[0] if ".lora_" in key else key.rsplit(".alpha", 1)[0] for key in all_keys})
+
+    entries = []
+    for name in bases:
+        a = lora[f"{name}.lora_A.weight"]
+        b = lora[f"{name}.lora_B.weight"]
+        alpha_key = f"{name}.alpha"
+        if alpha_key in lora:
+            scale = float(lora[alpha_key]) / a.shape[0]
+        else:
+            scale = 1.0
+        b_scaled = b * (scale * TURBO_LORA_STRENGTH)
+        entries.extend((key, a, b_part) for key, b_part in _targets(name, b_scaled, inner_dim))
+    return {
+        "label": f"{TURBO_LORA_REPO}/{TURBO_LORA_FILE}",
+        "scale": 1.0,  # scale already applied per-entry above
+        "entries": entries,
+    }
+
+
+def _apply(entries, params, sign: float) -> None:
+    """Fold (sign * scale * (B @ A)) into each target parameter in place."""
+    for key, a, b in entries:
+        param = params.get(key)
+        if param is None:
+            raise KeyError(f"LoRA target `{key}` not found in the transformer")
+        delta = sign * (b.to(torch.float32) @ a.to(torch.float32))
+        param.data = (param.data.float() + delta.to(param.device)).to(param.dtype)
+
+
+def apply_lora(transformer) -> str | None:
+    """Fold the wushu action LoRA — and, when enabled, the Turbo LoRA — into the transformer weights.
+
+    Returns a status line, or `None` when nothing could be folded.
+    """
+    inner_dim = transformer.config.num_attention_heads * transformer.config.attention_head_dim
+    params = dict(transformer.named_parameters())
+
+    loaded = []
+
+    try:
+        wushu = _load_wushu_lora(inner_dim)
+        _apply(wushu["entries"], params, wushu["scale"])
+        loaded.append(f"wushu-action ({wushu['label']}, {len(wushu['entries'])} weights, scale={wushu['scale']})")
+        print(f"[lora] wushu action LoRA folded: {len(wushu['entries'])} weights", flush=True)
+    except Exception as error:
+        print(f"[lora] WARNING: failed to load the wushu action LoRA: {error}", flush=True)
+
+    if USE_TURBO:
+        try:
+            turbo = _load_turbo_lora(inner_dim)
+            _apply(turbo["entries"], params, turbo["scale"])
+            loaded.append(f"turbo ({turbo['label']}, {len(turbo['entries'])} weights)")
+            print(f"[lora] turbo LoRA folded: {len(turbo['entries'])} weights", flush=True)
+        except Exception as error:
+            print(f"[lora] WARNING: failed to load the turbo LoRA: {error}", flush=True)
+
+    if not loaded:
+        return None
+
+    return "LoRAs folded: " + " + ".join(loaded)

h3_split_blocks.py

@@ -0,0 +1,138 @@
+"""The halves of a **split** MiniMax-H3 deployment, for both of its checkpoint partitions.
+
+MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so `MiniMaxH3Blocks` is cut
+at its `text_encoder` step: the 62.14 GiB Qwen3-VL runs in the conditioner Space, everything else in a generator
+Space, and `prompt_embeds` + `text_token_tags` is the whole wire format between them.
+
+`resize` / `setup` run on **both** sides: they own no pretrained component, and each half needs the canvas and the
+prepared keyframes or normalized references. Both conditioner halves also return the resolved `height` / `width` /
+`num_frames`, which the generating half pins rather than re-deriving.
+
+Two things the blocks leave to the caller: a keyframe reaches them EXIF-transposed and in RGB, and the `t2va` / `fl2va`
+frame count is aligned to `17 * n + 5` before the call, since that arithmetic lives on the denoising side of the cut.
+"""
+
+from diffusers.modular_pipelines.minimax_h3.before_encoder import MiniMaxH3Ref2VASetupStep
+from diffusers.modular_pipelines.minimax_h3.decoders import MiniMaxH3AfterDenoiseStep
+from diffusers.modular_pipelines.minimax_h3.encoders import (
+    MiniMaxH3Ref2VAReferenceEncoderStep,
+    MiniMaxH3Ref2VATextEncoderStep,
+    MiniMaxH3TextEncoderStep,
+)
+from diffusers.modular_pipelines.minimax_h3.modular_blocks_minimax_h3 import (
+    MiniMaxH3AutoKeyframeVaeEncoderStep,
+    MiniMaxH3AutoResizeStep,
+    MiniMaxH3CoreDenoiseStep,
+    MiniMaxH3DecodeStep,
+    MiniMaxH3Ref2VACoreDenoiseStep,
+    _generation_outputs,
+)
+from diffusers.modular_pipelines.modular_pipeline import SequentialPipelineBlocks
+from diffusers.modular_pipelines.modular_pipeline_utils import OutputParam
+
+
+def _wire_outputs(num_frames: bool = True) -> list[OutputParam]:
+    """The wire format of the split. `num_frames` is declared by the `ref2va` half alone, whose setup resolves one."""
+    return [
+        OutputParam.template("prompt_embeds"),
+        OutputParam("text_token_tags", description="The per-row modality tag of every row of `prompt_embeds`."),
+        OutputParam("height", type_hint=int, description="Resolved height of the generated video in pixels."),
+        OutputParam("width", type_hint=int, description="Resolved width of the generated video in pixels."),
+        *(
+            [OutputParam("num_frames", type_hint=int, description="Resolved number of frames, of the form 17 * n + 5.")]
+            if num_frames
+            else []
+        ),
+    ]
+
+
+class MiniMaxH3ConditionerBlocks(SequentialPipelineBlocks):
+    """The conditioner half of a split MiniMax-H3: the keyframes on the canvas plus the Qwen3-VL read at layer 50."""
+
+    model_name = "minimax-h3"
+    block_classes = [MiniMaxH3AutoResizeStep, MiniMaxH3TextEncoderStep]
+    block_names = ["resize", "text_encoder"]
+
+    @property
+    def description(self):
+        return (
+            "The conditioner half of a split MiniMax-H3 deployment: puts the keyframes onto the target canvas and "
+            "encodes MiniMax-H3's presentation of the request into the `prompt_embeds` / `text_token_tags` pair the "
+            "denoising half consumes. The frame count is the caller's to align."
+        )
+
+    @property
+    def outputs(self):
+        return _wire_outputs(num_frames=False)
+
+
+class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
+    """The denoising half of a split MiniMax-H3: `MiniMaxH3Blocks` with its `text_encoder` step removed."""
+
+    model_name = "minimax-h3"
+    block_classes = [
+        MiniMaxH3AutoResizeStep,
+        MiniMaxH3AutoKeyframeVaeEncoderStep,
+        MiniMaxH3CoreDenoiseStep,
+        MiniMaxH3AfterDenoiseStep,
+        MiniMaxH3DecodeStep,
+    ]
+    block_names = ["resize", "vae_encoder", "denoise", "after_denoise", "decode"]
+
+    @property
+    def description(self):
+        return (
+            "The denoising half of a split MiniMax-H3 deployment: the `t2va` / `fl2va` branch of `MiniMaxH3Blocks` "
+            "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
+            "62.14 GiB Qwen3-VL conditioner is never loaded here."
+        )
+
+    @property
+    def outputs(self):
+        return _generation_outputs()
+
+
+class MiniMaxH3Ref2VAConditionerBlocks(SequentialPipelineBlocks):
+    """The conditioner half of a split `ref2va`: the resolved plan plus the Qwen3-VL read at its 50th layer."""
+
+    model_name = "minimax-h3"
+    block_classes = [MiniMaxH3Ref2VASetupStep, MiniMaxH3Ref2VATextEncoderStep]
+    block_names = ["setup", "text_encoder"]
+
+    @property
+    def description(self):
+        return (
+            "The conditioner half of a split MiniMax-H3 `ref2va` deployment: resolves the request plan (canvas, frame "
+            "count, references normalized onto MiniMax-H3's own rates and resolutions) and encodes MiniMax-H3's "
+            "presentation of it into the `prompt_embeds` / `text_token_tags` pair the denoising half consumes."
+        )
+
+    @property
+    def outputs(self):
+        return _wire_outputs()
+
+
+class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
+    """The denoising half of a split `ref2va`: the `ref2va` branch with its `text_encoder` step removed."""
+
+    model_name = "minimax-h3"
+    block_classes = [
+        MiniMaxH3Ref2VASetupStep,
+        MiniMaxH3Ref2VAReferenceEncoderStep,
+        MiniMaxH3Ref2VACoreDenoiseStep,
+        MiniMaxH3AfterDenoiseStep,
+        MiniMaxH3DecodeStep,
+    ]
+    block_names = ["setup", "reference_encoder", "denoise", "after_denoise", "decode"]
+
+    @property
+    def description(self):
+        return (
+            "The denoising half of a split MiniMax-H3 `ref2va` deployment: the `ref2va` branch of `MiniMaxH3Blocks` "
+            "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
+            "62.14 GiB Qwen3-VL conditioner is never loaded here. The transformer is the `transformer_ref` partition."
+        )
+
+    @property
+    def outputs(self):
+        return _generation_outputs()

requirements.txt

@@ -0,0 +1,24 @@
+# diffusers is installed from the canonical MiniMax-H3 pull request,
+# https://github.com/huggingface/diffusers/pull/14371 ("Minimax h3 follow up (review & refactor)"), pinned to a
+# **commit** rather than to its `minimax-h3-refactor` branch: the PR is a WIP and its head moves, and this Space's
+# blocks subclass its block classes. Re-pin — and re-check `h3_split_blocks.py` against the block names of the new
+# head — whenever the PR updates.
+#
+#   665f578278365ea4a3318cb8c9b66ce6c01204b9 = refs/pull/14371/head at the time of this deploy
+--extra-index-url https://download.pytorch.org/whl/cu130
+diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
+torch==2.11.0
+torchvision==0.26.0
+# The Qwen3-VL processor decides the vision patch count, so a different minor changes the conditioning.
+transformers==5.8.0
+accelerate==1.14.0
+# diffusers pins <2.
+huggingface-hub==1.24.0
+# No `kernels` pin on purpose: the Hub attention backends want `kernels>=0.12.3`, and that version breaks
+# transformers 5.8.0 at import.
+# PyAV muxes the generated soundtrack onto the frames (`encode_video`).
+av
+pillow
+numpy
+requests
+safetensors>=0.8.0