README.md

metadata

title: MiniMax-H3 Reference
emoji: 🎭
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: true
short_description: Unquantized MiniMax-H3 from image, audio, video refs
suggested_hardware: zero-a10g

MiniMax-H3 — omni-references, unquantized, split across two Spaces

Joint video and soundtrack out of a single denoising pass, conditioned on an ordered list of image, video and audio references, at bfloat16 with no quantization anywhere.

This Space is the denoising half of the ref2va task: the 61.73 GiB transformer_ref partition and the two autoencoders. The 62.14 GiB Qwen3-VL conditioner runs in qwen3vl-conditioner, which this Space calls over the gradio API for every request — the same conditioner Space, and the same resident weights, that the keyframe half minimax-h3 uses.

Why split

MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage. An unquantized single Space is therefore impossible. Cut the ref2va branch of MiniMaxH3Blocks at its text_encoder step and both halves fit unquantized:

Space	Subfolders	Download	Resident
qwen3vl-conditioner	text_encoder/ + tokenizer/ + processor/	66.7 GB	62.15 GiB bf16
this one	transformer_ref/ + vae/ + audio_vae/	77.3 GB	61.73 GiB bf16 + 10.43 GiB float32

References

A request carries up to 12 references — at most 9 images, 3 videos and 3 audio clips — in the order the model reads them. The order is semantic: it numbers the labels of MiniMax-H3's prompt presentation (<Picture 1>, <Video 1>, <Audio 1>) and it advances the shared audio/video rotary clock, so the same references in a different order are a different request. This demo lays the slots out as one tab per modality in reading order — images, then audio, then video — and assembles the request that way. The Images tab opens with two slots and + Add another image reveals the rest, up to the model's own nine; the audio and video tabs hold one each. A reference left in a tab that is not the open one is still part of the request; the tabs lay the slots out, they do not choose between them.

Rules the model imposes, enforced here before anything is uploaded:

• an audio reference cannot be the only one; it needs an image or a video alongside it,
• a reference video runs 2 to 15 seconds, and brings its own soundtrack with it,
• the generated duration may be left to the references, but only when exactly one of them carries a soundtrack — which is why the duration slider disappears when a single reference can set it, and comes back when two can or when the one that could is out of range.

Example assets

examples/subject.png is a studio portrait by Oliver Dohrn cropped to head and shoulders — Pexels licence, free to use. examples/motion.mp4 is a synthetic clip from the parity fixtures, and examples/voice.wav is utterance 1462-170145-0022 of LibriSpeech dev-clean — CC BY 4.0, read from a public-domain LibriVox recording. It is 16 kHz mono on purpose: the audio VAE wants 32 kHz, so the example exercises the torchaudio resample the ref2va path needs.

How the split is expressed

MiniMaxH3Blocks is one SequentialPipelineBlocks whose branches are picked per request — and per workflow= — from the inputs, ref2va being the branch references selects:

setup -> text_encoder -> reference_encoder -> denoise -> after_denoise -> decode

where denoise is itself prepare_layout -> prepare_latents -> set_timesteps -> denoise, against the transformer_ref partition.

h3_split_blocks.py subclasses it with the text_encoder step removed. Dropping the step drops the three components it declares, so load_components resolves transformer_ref / vae / audio_vae / the two schedulers out of the shared modular_model_index.json and never fetches the conditioner — and prompt_embeds and text_token_tags become ordinary required inputs of the pipeline call:

pipe = MiniMaxH3Ref2VAGeneratorBlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
pipe.load_components(dtype=torch.bfloat16)
state = pipe(prompt_embeds=..., text_token_tags=..., references=[...], height=544, width=960, num_frames=124,
             num_inference_steps=28)

Only text encoding is remote. reference_encoder is the ref2va branch's own encoder step — it runs the video VAE over the image and video references and the audio VAE over the soundtracks, and it is where the references' latent geometry is resolved — so it stays on this side, next to the autoencoders the conditioner Space does not hold.

The wire format is the same two tensors as the keyframe half: (1, num_text_tokens, 5120) bfloat16 and (num_text_tokens,) int64, carried as one safetensors file with the resolved height / width / num_frames in its metadata header. What differs is only what the conditioner is shown, so the references travel to it as files: ref2va's presentation puts a vision block in front of the prompt for every image and every merged video frame pair. An audio reference contributes its "<Audio j>: " label and nothing else — a waveform never reaches the conditioner — but it still goes over, because a single audio-bearing reference is what resolves num_frames when the request leaves it open.

The setup step runs on both halves. It owns no component (PIL, PyAV-decoded media and arithmetic) and it resolves the canvas, the 17 * n + 5 frame count and the references prepared at their own resolutions. It is deterministic over the same files, and the conditioner returns the plan it resolved so this Space pins the same canvas and frame count rather than re-deriving them.

AoTI-compiled blocks

With H3_AOTI=1 the 50 repeated transformer blocks run from a compiled package, multimodalart/minimax-h3-aoti:bf16/torch2.11/sm120/dynamic — a single dynamic-sequence artifact that serves every canvas, duration, reference set and prompt length.

It is the same package the transformer/ partition runs, and nothing about it is partition-specific. The two config.json files are identical field for field, and the package carries no weights at all: LazyAOTIModel binds each block's own live state_dict() by name on its first forward. Patching it in is startup CPU work and costs no GPU time.

It removes a near-constant ~0.5 s/step — 50 blocks' worth of kernel-launch overhead plus the norm / rotary / AdaLN epilogues around the matmuls — and cannot touch the matmuls themselves, so it pays best where the block is not compute bound. ref2va packs the reference rows in front of the generated ones, which makes the sequence longer than a keyframe request at the same canvas and moves it further toward compute bound.

Nothing is paid for with GPU time

The 77.3 GB download and the load happen at startup: import spaces at module top patches torch.cuda before any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this Space's CPU. A @spaces.GPU call is therefore only the placement (once), the two reference encoders, the denoise loop and the two decoders.

One thing does not happen at startup: the move onto the card. spaces' startup torch.pack() writes every startup-resident CUDA tensor to a second copy on disk before deleting the downloaded originals, and 77.3 GB of weights plus a 77.3 GB pack is 154.6 GB against a 150 GB quota — the Space is evicted mid-pack with OSError: [Errno 28] No space left on device. Placement therefore happens on the first GPU call, PIPE.to("cuda") at the top of the @spaces.GPU function: about 10 s of PCIe once, then a no-op walk, and the denoise loop runs with everything resident and no offloading at all.

The references are decoded inside that call too, from their paths rather than as decoded media. A @spaces.GPU argument crosses a process boundary by pickling, and a 5 s 1344x768 reference video is 370 MB of frames once PyAV has expanded it.

Generation constraints

Fixed by the checkpoint: 24 fps, a 768 pixel short edge, 5 to 15 s, num_frames snapped up to the next 17 * n + 5, no CFG and no negative prompt (it is guidance-distilled, so every step is one forward pass). The duration slider stops at 14 s because it is the snapped count that has to hold for the ceiling: 15 s is 360 frames, which rounds up to 362, i.e. 15.083 s, and is refused.

GPU time is reserved per request, not per Space

MiniMax-H3 attends over one packed sequence, so what a step costs is a function of that sequence's length alone — and on this half the references dominate it. A single 1344x768 image reference is ~7168 conditioning rows plus the vision block it puts in front of the prompt; a 2.5 s video reference is another ~17000. The same 960x544, 124-frame request runs 2.4 s/step with no references and 16 s/step with an image and a video.

get_duration prices that before the call instead of reserving a flat ceiling for everything. It takes the arguments of the @spaces.GPU function, so it has the conditioner's own text_token_tags (exact) and the reference files (measured from metadata, no decode), and evaluates

S = text rows + reference rows + target rows
seconds = placement + reference encode + steps * (LINEAR * S + QUADRATIC * S**2) * SAFETY + decode + pad

fitted on the t2va half and checked against live ref2va requests to about 10%. It matters beyond tidiness: the pool reserves whatever number it is given, and a flat 900 s is what makes a busy account fail admission with "You have too many ZeroGPU credits allocated to running tasks." A typical single-image request now reserves ~460 s.

Every request carries the full placement allowance, because nothing on this side knows whether the worker it lands on is cold and a cold one pays the lazy 72.16 GiB PIPE.to("cuda") inside its first GPU call.

Space variables

Variable	Default	Meaning
H3_CONDITIONER	multimodalart/qwen3vl-conditioner	The public Space this one asks for embeddings; the client passes no token, so the call runs on the caller's own quota.
H3_MODEL_REPO	MiniMaxAI/MiniMax-H3	The diffusers-layout checkpoint. Public.
H3_AOTI	0	1 loads the compiled block package.
H3_PLACEMENT	lazy	lazy moves all 72.16 GiB onto the card on the first GPU call and leaves it there; offload hands placement to ComponentsManager.enable_auto_cpu_offload instead.
H3_ATTENTION	_native_cudnn	cuDNN's fused kernel, 10–20% faster than the SDPA default and needs nothing installed. flash-attention 3 is sm90-only and this pool is sm120. The two VAEs are pinned to torch SDPA instead: they are float32, which cuDNN has no kernel for.
H3_GPU_DURATION_MIN / _MAX	120 / 1500	Bounds on what get_duration may reserve.
H3_PLACEMENT_ALLOWANCE	90	Seconds of the reservation set aside for a cold worker's placement.
H3_GPU_SIZE	xlarge	ZeroGPU allocation size. large does not fit.

Whose GPU quota pays

Two cards are booked per request — this Space's denoise loop and the conditioner's forward — and both are billed to the requesting user, with nothing here arranging it: gradio_client attaches the caller's own x-ip-token to every outgoing call, reading it off gradio's LocalContext inside the event listener (Client.send_data -> add_zero_gpu_headers), and ZeroGPU charges the booking to whatever that token identifies.

A caller with no token to forward — a gradio_client script rather than a browser — leaves the conditioner's booking attributed to this Space's pod IP and its small shared quota. An unattributed caller may book at most 120 credits at a time and an xlarge booking costs twice its seconds, so the conditioner books the encode (45 s) and a prompt upsample (60 s) as two separate calls, each within that ceiling.

Secrets

None are required. Everything this Space downloads is public — the MiniMaxAI/MiniMax-H3 checkpoint and the multimodalart/minimax-h3-aoti packages — and the conditioner is a public Space called on the requesting user's own ZeroGPU token, never on an org token.

Where diffusers comes from

MiniMax-H3 is modular-only and not in a released diffusers, so requirements.txt installs it from the canonical pull request, huggingface/diffusers#14371, pinned to the commit 665f5782 (refs/pull/14371/head) rather than to the moving minimax-h3-refactor branch.

That PR is a WIP, so it needs re-pinning whenever it updates, and h3_split_blocks.py — which subclasses its block classes to cut the pipeline in two — has to be re-checked against the new head at the same time.

Two of those are ref2va-only and easy to miss. PyAV decodes a reference video or audio file as the reference is built, and torchaudio resamples a soundtrack that is not already at the audio VAE's 32 kHz — a 32 kHz reference skips the resample entirely, so the dependency only shows up once someone brings audio at another rate:

ImportError: Resampling a MiniMax-H3 reference soundtrack from 24000 Hz to 32000 Hz needs `torchaudio`.

The conditioner Space needs it as well: its setup step normalizes the very same waveforms this one does.