--- license: apache-2.0 library_name: transformers pipeline_tag: feature-extraction base_model: - Qwen/Qwen3-VL-8B-Instruct - Qwen/Qwen3-VL-32B-Instruct base_model_relation: merge datasets: - detection-datasets/coco - WenhaoWang/VidProM - tipi2v/TIP-I2V tags: - qwen3-vl - vision-language - minimax-h3 - image-to-video - custom-code - experimental --- # Qwen3-VL stitched 8B L16 to 32B L29 bridge (25M) MiniMax H3 uses an intermediate Qwen3-VL-32B hidden-state sequence as part of its image-to-video conditioning. This repository tests whether that sequence can be approximated without running the first 29 decoder blocks of Qwen3-VL-32B. The replacement runs the smaller Qwen3-VL-8B prefix, converts its hidden width from 4,096 to 5,120 with a learned stateful bridge, and resumes the original 32B computation at block 30. The result is an experimental **H3 conditioning frontend**, not a general-purpose Qwen model. It accepts image-and-text inputs and returns the raw, unnormalized hidden state after the 50th 32B decoder block. It has no language-model head and cannot generate text. ## Paired H3 previews Each animation places the MiniMax NVFP4 control on the left and the stitched frontend on the right. Within a pair, the source image, prompt, generation seed, sampler, schedule, and decoder are held constant. | “Make him use the anvil” | “The cat's paws are making a move” | |---|---| | ![MiniMax and stitched anvil videos](images/anvil-divergence.webp) | ![MiniMax and stitched cat-paw videos](images/cat-paw-divergence.webp) | These are two of ten held-out TIP-I2V comparisons. They are deliberately larger-divergence cases, not cherry-picked best outputs; aggregate measurements and limitations appear under [Evaluation summary](#evaluation-summary). ## Architecture Layer labels in this repository count completed decoder blocks. `L16` is the state after 8B block indices 0–15, `L29` is the state after 32B block indices 0–28, and `L50` is the state after 32B block indices 0–49. The packaged 32B tail therefore contains original block indices 29–49. ```text image and text -> Qwen3-VL-8B vision tower, token embedding, and blocks 0-15 -> raw post-L16 state [batch, tokens, 4096] -> frozen affine map + four causal width-512 Gated DeltaNet residual lanes -> predicted post-L29 state [batch, tokens, 5120] -> Qwen3-VL-32B blocks 29-49 -> raw post-L50 H3 conditioning [batch, tokens, 5120] ``` The bridge predicts ```text affine(source) * exp(bounded_causal_log_gain(source_history)) + causal_dense_residual(source_history) ``` The affine map provides the strong token-local baseline. The recurrent lanes correct sequence-dependent differences, including changes in which token positions carry a large share of the activation magnitude. Both Qwen prefixes/tails and the affine baseline were frozen during the final bridge training. | Artifact component | BF16 elements | Role | |---|---:|---| | Trainable recurrent, gain, and residual parameters | 25,226,801 | Learned bridge correction | | Frozen affine weight and bias | 20,976,640 | 4,096-to-5,120 baseline | | Complete standalone bridge | 46,203,441 | About 88.1 MiB of tensor data | | Fully repacked H3 frontend | 14,571,629,601 | About 27.14 GiB of tensor data | ## Intended use The supported use is producing an experimental substitute for MiniMax H3's Qwen L50 conditioning under the raw image/text presentation used during training and evaluation. The release is useful for reproducing the reported experiment, generating candidate H3 conditioning tensors, and studying intermediate-state transfer between related vision-language models. The release is not intended as: - a drop-in Qwen3-VL-32B chat or text-generation model; - an exact reconstruction of Qwen3-VL-32B hidden states; - a demonstrated replacement for the audio-conditioning path; or - a quality or safety benchmark for MiniMax H3. ## Repository contents - `model-00001-of-00008.safetensors` through `model-00008-of-00008.safetensors`, plus `model.safetensors.index.json`: the complete BF16 stitched frontend, loadable through `transformers.AutoModel`. - `info/assets/bridge.safetensors`: the promoted bridge alone. Using it requires separate 8B and 32B checkpoints and compatible splice code. - `examples/create_h3_conditioning.py`: an end-to-end image/prompt example that writes L50 state, MiniMax token tags, and input token IDs. - `info/`: the experiment rationale, architecture selection, training protocol, evaluation interpretation, and compact machine-readable results. - `images/`: three paired animated previews from the held-out release evaluation. The standalone promoted bridge has SHA-256 `515856a5a53361ae575dde1943338b761c0d09040d67ae51e843ffacc42ec338`. The per-shard checksums and export provenance are in [`export-report.json`](export-report.json). The independent full-file and tensor-layout audit is in [`info/results/checkpoint-audit.json`](info/results/checkpoint-audit.json). After cloning all Git LFS objects, the published shards can be verified with: ```bash uv run python info/code/audit_stitched_checkpoint.py . \ --output info/results/checkpoint-audit.json ``` ## Installation The recurrent bridge uses Flash Linear Attention (FLA) 0.5.1. For a normal CUDA environment: ```bash git clone https://huggingface.co/YOUR_NAME/qwen3-vl-25m-stitched-8bL16-to-32bL29 cd qwen3-vl-25m-stitched-8bL16-to-32bL29 uv sync --extra sequence ``` The processor also requires a `torchvision` build compatible with the installed PyTorch build. The project dependency resolver installs this for a normal environment. Managed GPU images often include a working PyTorch/CUDA/Triton combination that should not be replaced. One tested setup procedure is: ```bash uv venv --system-site-packages uv pip install --no-deps fla-core==0.5.1 flash-linear-attention==0.5.1 uv pip install einops==0.8.1 transformers==5.15.0 safetensors==0.8.0 accelerate==1.14.0 ``` Verify that the image already provides a matching `torchvision`; the H3 test images used PyTorch 2.8.0 with torchvision 0.23.0. On H100/H200, FLA 0.5.1 also needs `tilelang==0.1.13` for the guarded Hopper backward kernel. In an inference-only environment this dependency may still be imported by the installed FLA path, so validate the exact environment before a production run. ## Creating an H3 conditioning tensor The training and release evaluations used MiniMax's raw FL2VA-style presentation rather than a Qwen chat template. For one image, the processor input is conceptually: ```text : <|vision_start|><|image_pad|><|vision_end|>PROMPT ``` There is no added BOS/chat wrapper. The example resizes the source image to the generation canvas before Qwen tokenization, because the canvas determines the vision-token grid. Its defaults match the 608×352 release evaluation: ```bash uv run python examples/create_h3_conditioning.py \ --model . \ --image /path/to/input.png \ --prompt "The subject turns toward the camera" \ --output conditioning.safetensors ``` The output contains: - `layer_50`: BF16 state with shape `[1, sequence_length, 5120]`; - `minimax_token_tags`: `0` for the complete vision block and `1` for text positions; and - `input_ids`: the expanded Qwen token sequence used for the forward pass. Because the architecture is implemented as custom Transformers code, Hub loading requires `trust_remote_code=True`. Pin a reviewed Hub commit when loading code from a remote repository. ```python import torch from transformers import AutoModel, AutoProcessor model_id = "YOUR_NAME/qwen3-vl-25m-stitched-8bL16-to-32bL29" processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) model = AutoModel.from_pretrained( model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa", ).eval() # Construct `inputs` with the same raw presentation and image canvas described above. with torch.inference_mode(): layer_50 = model(**inputs).last_hidden_state ``` ## Evaluation summary Two paths are distinguished throughout the reports: - **reference path:** vanilla BF16 Qwen3-VL-32B through raw post-L50; - **stitched path:** Qwen3-VL-8B through post-L16, the bridge, then 32B blocks 29–49. For next-token evaluation only, the same frozen 32B blocks 50–63 and language-model head were attached after each L50 state. Those evaluation-only layers are not included in this release. The selected checkpoint was produced by epoch 6 of a seven-epoch online training campaign. On its fixed 20-case model-selection gate it reached mean Top-5 intersection `4.05 / 5` and reference-to-stitched KL divergence `0.5859` nats. A separate ten-case TIP-I2V release set was not used for optimization or model selection. Images were first resized to the exact 608×352 H3 generation canvas. Against the BF16 reference path: | Next-token metric | Result | |---|---:| | Exact Top-1 agreement | 7 / 10 | | Mean Top-5 intersection | 3.5 / 5 | | Mean Top-25 intersection | 19.4 / 25 | | Mean Top-100 intersection | 80.2 / 100 | | Full-vocabulary logit cosine | 0.9309 | | Reference-to-stitched KL divergence | 0.2019 nats | The same ten cases were rendered as fixed-seed MiniMax H3 pairs. The control used MiniMax's shipped NVFP4 frontend; the experimental member used the stitched frontend. All other recorded generation settings were held constant. | Paired-video metric | Result | |---|---:| | Mean framewise SSIM | 0.8018 | | Median framewise SSIM | 0.8423 | | Mean PSNR | 22.857 dB | These metrics measure pixel-trajectory similarity between paired outputs, not prompt adherence or human preference. The sample is too small to establish a general video-quality result. The two paired previews near the top of this card intentionally show larger-divergence cases and should be interpreted alongside the aggregate metrics. The complete preview index also identifies a close pair and is available in [`images/README.md`](images/README.md). ## Limitations - General Qwen chat behavior was not an optimization target. A separate chat-format probe had weak next-token agreement even when the raw H3-format probe was strong. - The bridge is causal and stateful. Resetting its recurrence at every token removes most of the measured benefit; independent token-by-token calls are not equivalent to a full-sequence call. - The model supports Qwen's 262,144-token domain, but training did not exhaustively exercise that limit. The longest realized context in the final seven-epoch campaign was 12,188 tokens; an earlier dedicated multi-image phase reached 6,451 tokens. - L50 hidden-state magnitude can differ substantially even when paired H3 video output remains plausible. Hidden-state cosine or L2 alone was not a reliable accept/reject criterion in the ten-case video sample. - The custom loader was structurally and synthetically validated with Transformers 5.15 and FLA 0.5.1. GPU-backed validation used the equivalent bridge and splice components during training and H3 generation; the newly repacked eight-shard `AutoModel` artifact still requires a checkpoint- backed GPU smoke test as a single packaged loader. ## Detailed reports - [Documentation index and terminology](info/README.md) - [Why an affine-only map was insufficient](info/affine_failure.md) - [Architecture search and the four-lane selection](info/architecture_hunt.md) - [Training data, objectives, length sampling, and promotion](info/training_methodology.md) - [Evaluation protocol, units, and interpretation](info/evaluation.md) ## Data sources | Dataset | Role | Included in this repository? | |---|---|---:| | [COCO 2017](https://cocodataset.org/#download) | Training images and preliminary probes | No | | [WenhaoWang/VidProM](https://huggingface.co/datasets/WenhaoWang/VidProM) | Training prompt source | No | | [tipi2v/TIP-I2V](https://huggingface.co/datasets/tipi2v/TIP-I2V) | Held-out evaluation and paired-video source | No | COCO images and VidProM prompts were combined into synthetic multimodal training sequences; they were not treated as semantically paired captions. TIP-I2V was held out from optimization and model selection. It appears in the Hub metadata so that the evaluation source is discoverable, not because it was used to train the bridge. ## License and attribution Original bridge code and bridge weights are released under Apache-2.0. The repacked artifact also contains weights from the Apache-2.0 Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct checkpoints. Training used COCO 2017 images and VidProM prompt records; TIP-I2V supplied held-out evaluation records and preview media. VidProM and TIP-I2V are distributed under CC-BY-NC-4.0. No source dataset records are redistributed here, and this repository's Apache-2.0 license does not relicense upstream models, datasets, or evaluation media. See [`NOTICE`](NOTICE) for third-party acknowledgements.