Lance-3B-Alis-MLX-Traced

Part of the Lance · Alis MLX collection.

ByteDance Lance 3B (image + video) converted to Apple MLX, byte-clean against the original PyTorch.

Layout

Two standalone weights, one per variant — matching ByteDance's Lance_3B/ + Lance_3B_Video/ layout:

Path	Variant	Size	dtype	keys
Lance_3B/model.safetensors	image (LLM + adapters)	24.7 GB	F32	1021
Lance_3B_Video/model.safetensors	video (standalone: backbone + 31-frame pos-embed + video ViT)	28.4 GB	F32	1411

The image ViT and the Wan 2.2 VAE are separate files (see the repo for setup).

The weight is not the point — the verification is

The image weight (Lance_3B/) is bit-identical (SHA256 5ede2f0a…547817) with RockTalk/Lance-3B-MLX — both are the same deterministic F32 conversion from bytedance-research/Lance. (This is the F32 build; a separate bf16 build is mlx-community/Lance-3B-bf16.)

This weight is byte-identical to an existing MLX port (RockTalk's Lance-3B-MLX), so the point of this release is not the weight but its public verification trace: every stage of the port was cross-validated against the original PyTorch via byte-diff before the next stage started, and the full harness + lesson log is public. We don't claim to be the first or only MLX port of Lance — other independent ports exist (e.g. mlx-community/lance-mlx); this is our own independent port with an openly published byte-diff trail:

👉 github.com/avlp12/lance_alis_mlx

Post-release correction (Stage 11, 2026-06-08). Two bugs were found and fixed after release — a ViT patch-token-order bug (x2t / image_edit / x2t_video) and an x2t_video temporal-mRoPE multiplier. Both were in the MLX pipeline / harness, not in these weights (SHA256 unchanged; t2i / t2v unaffected). Stage 7's PT-import gates had stayed blind because they fed PT our intermediates instead of recomputing from raw inputs; Stage 11 re-verifies non-blind (patches + positions PT-recomputed byte-identical, K = 8 top-1 8/8, min logit cos 0.999124 image / 0.999437 video). See the repo README's Post-release correction section.

Verification

Every gate uses original PyTorch direct import (not a clean re-implementation) under a shim, same PRNG (NumPy) on both sides, byte-diff at every layer. 25 lessons across stages 1–9 + Stage 11; see the repo's LEARNING_LOG/.

Stage	Component	Gate
1	PT → MLX weight conversion	bit-exact (SHA256 match)
5	Wan 2.2 VAE image path (T=1)	~40 dB PSNR round-trip vs PT
6	Flow matching + CFG (T2I)	end-to-end cos ≥ 0.999 vs PT 30-step
7	ViT + X→T + TI2I	forward parity; ⚠ preprocessing bug found post-release (see note)
8	3D causal video VAE	4 gates cos = 1.000000 (encode + decode)
9	T2V (video DiT + flow matching)	30-step latent cos ≥ 0.999, video pixel cos = 0.999338 vs PT
11	x2t image + video re-verification	patches + positions PT-recomputed byte-identical; K=8 top-1 8/8, min cos 0.999124 / 0.999437

Key gates only (stages 2–4 omitted for brevity; Stage 10 — quantization — is planned, not yet started, hence 9 → 11). Stage 11 is the post-release non-blind re-verification — x2t shown above; image_edit and video_edit are covered in the honesty note below.

The video weight reproduces STAGE 9 t2v exactly (single-step cos = 0.999916 / 0.999848 / 0.999452 vs PT). Its converted video supplement is 391/391 byte-identical to RockTalk's, which is in turn byte-clean vs the original PT supplement.

Honesty note. T2V is verified end-to-end and uses only the 1021-key subset. The video weight also bundles the video ViT (vit_model, byte-clean vs PT). x2t_video is implemented and re-verified non-blind in Stage 11 (patches + positions PT-recomputed byte-identical, K = 8 top-1 8/8, min logit cos 0.999437), under the scope "given identical resized / normalized frames and grid" — PT's real vit_transform bucket resize is out of scope. video_edit is now implemented and re-verified non-blind too (it is image_edit's method on the video path): PT recomputes patches, the video ViT, and the 3-slab mRoPE positions byte-identical; the 3-component velocity matches at min cos 0.99991 (raster control collapses v_full to 0.994); a 5-step trajectory holds at latent cos 0.999999; the cond VAE-encode scale is de-blind against a PT Wan VAE at cos 1.000000. image_edit's 3-component CFG velocity is now re-verified non-blind as well (min cos 0.999640, raster control collapses v_full). All six Lance tasks are implemented and verified; full pixel decode for the video paths is implied by the per-step + accumulation cos and the byte-clean Stage 8 VAE decode, not measured directly.

Usage

Inference is pure MLX — no PyTorch at runtime (PyTorch is imported only by the verification harnesses in tools/).

git clone https://github.com/avlp12/lance_alis_mlx
cd lance_alis_mlx
python3.12 -m venv .venv && source .venv/bin/activate
pip install mlx mlx-vlm transformers safetensors einops pillow huggingface_hub numpy

# weights: image (Lance_3B/) + video (Lance_3B_Video/)
hf download avlp12/Lance-3B-Alis-MLX-Traced --local-dir checkpoints/Lance-Alis
hf download RockTalk/Wan2.2-VAE-MLX --local-dir checkpoints/Wan2.2-VAE-MLX

# generate (see the repo README for the exact checkpoints/ layout)
PYTHONPATH=. .venv/bin/python tools/stage6_t2i_smoke.py    # text-to-image
PYTHONPATH=. .venv/bin/python tools/stage7_ti2i_smoke.py   # image edit

A Gradio web UI (app.py in the repo) wraps all six tasks for the browser (pip install gradio && python app.py).

Practical settings (measured). Lance images are trained at 768 (image_768res) — below 512 the output is out-of-distribution and breaks; video runs at a **480** preset. Decode must un-normalize generated latents with the Wan-VAE (mean, 1/std) scale (else it oversaturates). Lance is very prompt-sensitive — detailed prompts help a lot.

Quality (honest). Lance-3B is a 3B unified any-to-any model; its image / video generation is modest — clearly below dedicated diffusion engines (SDXL / FLUX). Its strengths are breadth (six tasks in one model) and the understanding / editing paths, not photorealism. (PT and our MLX produce the same modest 768 quality; the showcase images are best-case.)

Apple Silicon required (developed on M3 Ultra). Python 3.12.

License & citation

Apache 2.0 — same as upstream ByteDance Lance and Alibaba Wan 2.2 VAE.

@misc{fu2026lanceunifiedmultimodalmodeling,
      title         = {Lance: Unified Multimodal Modeling by Multi-Task Synergy},
      author        = {Fengyi Fu and Mengqi Huang and Shaojin Wu and Yunsheng Jiang and Yufei Huo and Hao Li and Yinghang Song and Fei Ding and Jianzhu Guo and Qian He and Zheren Fu and Zhendong Mao and Yongdong Zhang},
      year          = {2026},
      eprint        = {2605.18678},
      archivePrefix = {arXiv},
      primaryClass  = {cs.CV},
      url           = {https://arxiv.org/abs/2605.18678},
}

Acknowledgments

• ByteDance Lance team — original PyTorch model and research
• RockTalk — MLX checkpoint conversion used as the F32 parity reference (image + video supplement)
• Alibaba Wan 2.2 team — 3D Causal VAE architecture