valarauca1/qwen3-vl-25m-stitched-8bL16-to-32bL29

Qwen3-VL stitched 8B L16 to 32B L29 bridge (25M)

MiniMax H3 uses an intermediate Qwen3-VL-32B hidden-state sequence as part of its image-to-video conditioning. This repository tests whether that sequence can be approximated without running the first 29 decoder blocks of Qwen3-VL-32B. The replacement runs the smaller Qwen3-VL-8B prefix, converts its hidden width from 4,096 to 5,120 with a learned stateful bridge, and resumes the original 32B computation at block 30.

The result is an experimental H3 conditioning frontend, not a general-purpose Qwen model. It accepts image-and-text inputs and returns the raw, unnormalized hidden state after the 50th 32B decoder block. It has no language-model head and cannot generate text.

Paired H3 previews

Each animation places the MiniMax NVFP4 control on the left and the stitched frontend on the right. Within a pair, the source image, prompt, generation seed, sampler, schedule, and decoder are held constant.

“Make him use the anvil”	“The cat's paws are making a move”

These are two of ten held-out TIP-I2V comparisons. They are deliberately larger-divergence cases, not cherry-picked best outputs; aggregate measurements and limitations appear under Evaluation summary.

Architecture

Layer labels in this repository count completed decoder blocks. L16 is the state after 8B block indices 0–15, L29 is the state after 32B block indices 0–28, and L50 is the state after 32B block indices 0–49. The packaged 32B tail therefore contains original block indices 29–49.

image and text
    -> Qwen3-VL-8B vision tower, token embedding, and blocks 0-15
    -> raw post-L16 state [batch, tokens, 4096]
    -> frozen affine map + four causal width-512 Gated DeltaNet residual lanes
    -> predicted post-L29 state [batch, tokens, 5120]
    -> Qwen3-VL-32B blocks 29-49
    -> raw post-L50 H3 conditioning [batch, tokens, 5120]

The bridge predicts

affine(source) * exp(bounded_causal_log_gain(source_history))
    + causal_dense_residual(source_history)

The affine map provides the strong token-local baseline. The recurrent lanes correct sequence-dependent differences, including changes in which token positions carry a large share of the activation magnitude. Both Qwen prefixes/tails and the affine baseline were frozen during the final bridge training.

Artifact component	BF16 elements	Role
Trainable recurrent, gain, and residual parameters	25,226,801	Learned bridge correction
Frozen affine weight and bias	20,976,640	4,096-to-5,120 baseline
Complete standalone bridge	46,203,441	About 88.1 MiB of tensor data
Fully repacked H3 frontend	14,571,629,601	About 27.14 GiB of tensor data

Intended use

The supported use is producing an experimental substitute for MiniMax H3's Qwen L50 conditioning under the raw image/text presentation used during training and evaluation. The release is useful for reproducing the reported experiment, generating candidate H3 conditioning tensors, and studying intermediate-state transfer between related vision-language models.

The release is not intended as:

• a drop-in Qwen3-VL-32B chat or text-generation model;
• an exact reconstruction of Qwen3-VL-32B hidden states;
• a demonstrated replacement for the audio-conditioning path; or
• a quality or safety benchmark for MiniMax H3.

Repository contents

• model-00001-of-00008.safetensors through model-00008-of-00008.safetensors, plus model.safetensors.index.json: the complete BF16 stitched frontend, loadable through transformers.AutoModel.
• info/assets/bridge.safetensors: the promoted bridge alone. Using it requires separate 8B and 32B checkpoints and compatible splice code.
• examples/create_h3_conditioning.py: an end-to-end image/prompt example that writes L50 state, MiniMax token tags, and input token IDs.
• info/: the experiment rationale, architecture selection, training protocol, evaluation interpretation, and compact machine-readable results.
• images/: three paired animated previews from the held-out release evaluation.

The standalone promoted bridge has SHA-256 515856a5a53361ae575dde1943338b761c0d09040d67ae51e843ffacc42ec338.

The per-shard checksums and export provenance are in export-report.json. The independent full-file and tensor-layout audit is in info/results/checkpoint-audit.json.

After cloning all Git LFS objects, the published shards can be verified with:

uv run python info/code/audit_stitched_checkpoint.py . \
  --output info/results/checkpoint-audit.json

Installation

The recurrent bridge uses Flash Linear Attention (FLA) 0.5.1. For a normal CUDA environment:

git clone https://huggingface.co/YOUR_NAME/qwen3-vl-25m-stitched-8bL16-to-32bL29
cd qwen3-vl-25m-stitched-8bL16-to-32bL29
uv sync --extra sequence

The processor also requires a torchvision build compatible with the installed PyTorch build. The project dependency resolver installs this for a normal environment.

Managed GPU images often include a working PyTorch/CUDA/Triton combination that should not be replaced. One tested setup procedure is:

uv venv --system-site-packages
uv pip install --no-deps fla-core==0.5.1 flash-linear-attention==0.5.1
uv pip install einops==0.8.1 transformers==5.15.0 safetensors==0.8.0 accelerate==1.14.0

Verify that the image already provides a matching torchvision; the H3 test images used PyTorch 2.8.0 with torchvision 0.23.0. On H100/H200, FLA 0.5.1 also needs tilelang==0.1.13 for the guarded Hopper backward kernel. In an inference-only environment this dependency may still be imported by the installed FLA path, so validate the exact environment before a production run.

Creating an H3 conditioning tensor

The training and release evaluations used MiniMax's raw FL2VA-style presentation rather than a Qwen chat template. For one image, the processor input is conceptually:

<Picture 1>: <|vision_start|><|image_pad|><|vision_end|>PROMPT

There is no added BOS/chat wrapper. The example resizes the source image to the generation canvas before Qwen tokenization, because the canvas determines the vision-token grid. Its defaults match the 608×352 release evaluation:

uv run python examples/create_h3_conditioning.py \
  --model . \
  --image /path/to/input.png \
  --prompt "The subject turns toward the camera" \
  --output conditioning.safetensors

The output contains:

• layer_50: BF16 state with shape [1, sequence_length, 5120];
• minimax_token_tags: 0 for the complete vision block and 1 for text positions; and
• input_ids: the expanded Qwen token sequence used for the forward pass.

Because the architecture is implemented as custom Transformers code, Hub loading requires trust_remote_code=True. Pin a reviewed Hub commit when loading code from a remote repository.

import torch
from transformers import AutoModel, AutoProcessor

model_id = "YOUR_NAME/qwen3-vl-25m-stitched-8bL16-to-32bL29"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
).eval()

# Construct `inputs` with the same raw presentation and image canvas described above.
with torch.inference_mode():
    layer_50 = model(**inputs).last_hidden_state

Evaluation summary

Two paths are distinguished throughout the reports:

• reference path: vanilla BF16 Qwen3-VL-32B through raw post-L50;
• stitched path: Qwen3-VL-8B through post-L16, the bridge, then 32B blocks 29–49.

For next-token evaluation only, the same frozen 32B blocks 50–63 and language-model head were attached after each L50 state. Those evaluation-only layers are not included in this release.

The selected checkpoint was produced by epoch 6 of a seven-epoch online training campaign. On its fixed 20-case model-selection gate it reached mean Top-5 intersection 4.05 / 5 and reference-to-stitched KL divergence 0.5859 nats.

A separate ten-case TIP-I2V release set was not used for optimization or model selection. Images were first resized to the exact 608×352 H3 generation canvas. Against the BF16 reference path:

Next-token metric	Result
Exact Top-1 agreement	7 / 10
Mean Top-5 intersection	3.5 / 5
Mean Top-25 intersection	19.4 / 25
Mean Top-100 intersection	80.2 / 100
Full-vocabulary logit cosine	0.9309
Reference-to-stitched KL divergence	0.2019 nats

The same ten cases were rendered as fixed-seed MiniMax H3 pairs. The control used MiniMax's shipped NVFP4 frontend; the experimental member used the stitched frontend. All other recorded generation settings were held constant.

Paired-video metric	Result
Mean framewise SSIM	0.8018
Median framewise SSIM	0.8423
Mean PSNR	22.857 dB

These metrics measure pixel-trajectory similarity between paired outputs, not prompt adherence or human preference. The sample is too small to establish a general video-quality result.

The two paired previews near the top of this card intentionally show larger-divergence cases and should be interpreted alongside the aggregate metrics. The complete preview index also identifies a close pair and is available in images/README.md.

Limitations

• General Qwen chat behavior was not an optimization target. A separate chat-format probe had weak next-token agreement even when the raw H3-format probe was strong.
• The bridge is causal and stateful. Resetting its recurrence at every token removes most of the measured benefit; independent token-by-token calls are not equivalent to a full-sequence call.
• The model supports Qwen's 262,144-token domain, but training did not exhaustively exercise that limit. The longest realized context in the final seven-epoch campaign was 12,188 tokens; an earlier dedicated multi-image phase reached 6,451 tokens.
• L50 hidden-state magnitude can differ substantially even when paired H3 video output remains plausible. Hidden-state cosine or L2 alone was not a reliable accept/reject criterion in the ten-case video sample.
• The custom loader was structurally and synthetically validated with Transformers 5.15 and FLA 0.5.1. GPU-backed validation used the equivalent bridge and splice components during training and H3 generation; the newly repacked eight-shard AutoModel artifact still requires a checkpoint- backed GPU smoke test as a single packaged loader.

Detailed reports

• Documentation index and terminology
• Why an affine-only map was insufficient
• Architecture search and the four-lane selection
• Training data, objectives, length sampling, and promotion
• Evaluation protocol, units, and interpretation

Data sources

Dataset	Role	Included in this repository?
COCO 2017	Training images and preliminary probes	No
WenhaoWang/VidProM	Training prompt source	No
tipi2v/TIP-I2V	Held-out evaluation and paired-video source	No

COCO images and VidProM prompts were combined into synthetic multimodal training sequences; they were not treated as semantically paired captions. TIP-I2V was held out from optimization and model selection. It appears in the Hub metadata so that the evaluation source is discoverable, not because it was used to train the bridge.

License and attribution

Original bridge code and bridge weights are released under Apache-2.0. The repacked artifact also contains weights from the Apache-2.0 Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct checkpoints. Training used COCO 2017 images and VidProM prompt records; TIP-I2V supplied held-out evaluation records and preview media. VidProM and TIP-I2V are distributed under CC-BY-NC-4.0. No source dataset records are redistributed here, and this repository's Apache-2.0 license does not relicense upstream models, datasets, or evaluation media.

See NOTICE for third-party acknowledgements.