Unified Visual Representation and Scalable Multimodal Modeling
HF Paper · arXiv · Code · Project Page
UniSpace is a unified visual representation framework spanning image understanding, reconstruction, generation, and instruction-based editing. This repository provides the selected inference checkpoints and normalization statistics associated with the paper. ## Overview UniSpace contains two related components: 1. **Patch-reparameterized vision encoders.** PR-SigLIP2, PR-DINOv2, and PR-Qwen-ViT retain the semantic representation of pretrained vision encoders while adding the image detail needed for faithful reconstruction and high-quality generation. 2. **The UniSpace multimodal model.** A Qwen-based patch-reparameterized visual tokenizer is coupled with a Qwen3-8B Mixture-of-Transformers model for visual understanding, text-to-image generation, and image editing. The public code release is inference and evaluation only. UniSpace training code, optimizer states, private data pipelines, and cluster-specific files are not included. ## Checkpoints | Artifact | Role | Used for | |---|---|---| | `encoders/pr-siglip2-tokenizer.pt` | Unified PR-SigLIP2 tokenizer | Reconstruction and ImageNet generation | | `encoders/pr-dinov2-tokenizer.pt` | PR-DINOv2 tokenizer | Reconstruction and generation | | `encoders/pr-qwen-vit-tokenizer.pt` | PR-Qwen-ViT tokenizer | Reconstruction and UniSpace initialization | | `encoders/pr-siglip2-dit.pt` | Class-conditional DiT | PR-SigLIP2 ImageNet generation | | `encoders/pr-dinov2-dit.pt` | Class-conditional DiT | PR-DINOv2 ImageNet generation | | `stats/*.pt` | Latent normalization statistics | Matching encoder evaluation configs | | `unispace-sft-0012000/model-*.safetensors` | Final selected UniSpace SFT checkpoint (17 standard shards) | Understanding, generation, and editing | The selected UniSpace model is SFT step `0012000`, initialized from stage-3 step `0060000`. Earlier training checkpoints are not required for inference. File digests are provided in [`SHA256SUMS`](./SHA256SUMS). ## Download Download the complete release snapshot: ```bash hf download yjb6/UniSpace --local-dir checkpoints/UniSpace ``` Or download one artifact: ```bash hf download yjb6/UniSpace \ encoders/pr-dinov2-tokenizer.pt \ --local-dir checkpoints/UniSpace ``` Expected layout: ```text checkpoints/UniSpace/ ├── encoders/ │ ├── pr-siglip2-tokenizer.pt │ ├── pr-dinov2-tokenizer.pt │ ├── pr-qwen-vit-tokenizer.pt │ ├── pr-siglip2-dit.pt │ └── pr-dinov2-dit.pt ├── stats/ │ ├── pr-siglip2-normalization-stats.pt │ ├── pr-dinov2-normalization-stats.pt │ └── pr-qwen-vit-normalization-stats.pt └── unispace-sft-0012000/ ├── model.safetensors.index.json └── model-00001-of-00017.safetensors ... model-00017-of-00017.safetensors ``` ## Installation ```bash git clone https://github.com/yjb6/UniSpace.git cd UniSpace conda env create -f environment.yml conda activate rae hf download yjb6/UniSpace --local-dir checkpoints/UniSpace ``` The verified environment uses Python 3.10, PyTorch 2.8.0, torchvision 0.23.0, Transformers 4.57.3, and Accelerate 1.12.0. See the [GitHub reproduction guide](https://github.com/yjb6/UniSpace#installation) for dataset preparation, model dependencies, distributed launch commands, and canonical evaluator setup. ## Reproduce encoder reconstruction Prepare ImageNet-1K in its standard class-directory layout and set the paths described in the code repository. For example: ```bash cd patch-reparameterization bash run_eval_only.sh configs/release/pr-siglip2-imagenet256.yaml \ ../checkpoints/UniSpace/encoders/pr-siglip2-tokenizer.pt \ --output-dir ../outputs/pr-siglip2-recon \ --num-samples 50000 --batch-size 32 --no-zeroshot bash run_eval_only.sh configs/release/pr-dinov2-imagenet256.yaml \ ../checkpoints/UniSpace/encoders/pr-dinov2-tokenizer.pt \ --output-dir ../outputs/pr-dinov2-recon \ --num-samples 50000 --batch-size 64 --no-zeroshot bash run_eval_only.sh configs/release/pr-qwen-vit-imagenet256.yaml \ ../checkpoints/UniSpace/encoders/pr-qwen-vit-tokenizer.pt \ --output-dir ../outputs/pr-qwen-vit-recon \ --num-samples 50000 --batch-size 16 --no-zeroshot ``` Each run evaluates EMA weights and reports PSNR, SSIM, rFID, and sample count. The complete generation and multimodal benchmark commands are documented in the [main README](https://github.com/yjb6/UniSpace#reproduce-imagenet-generation). ## Results ### Patch-reparameterized encoders: reconstruction ImageNet-1K validation, 256 × 256, 50,000 images: | Encoder | PSNR ↑ | SSIM ↑ | rFID ↓ | |---|---:|---:|---:| | PR-SigLIP2 | 29.64 | 0.87 | 0.18 | | PR-DINOv2 | 30.84 | 0.90 | 0.14 | | PR-Qwen-ViT | 30.16 | 0.88 | 0.17 | ### Patch-reparameterized encoders: ImageNet generation ImageNet-1K class-conditional generation, 256 × 256, 50,000 images: | Encoder | CFG | gFID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ | |---|---:|---:|---:|---:|---:|---:| | PR-SigLIP2 | 1.0 | 4.409 | 6.675 | 190.66 | 0.722 | 0.646 | | PR-SigLIP2 | 1.2 | 2.799 | 6.055 | 248.05 | 0.777 | 0.611 | | PR-DINOv2 | 1.0 | 2.100 | 5.377 | 216.97 | 0.779 | 0.637 | | PR-DINOv2 | 1.2 | 1.877 | 4.888 | 274.16 | 0.822 | 0.605 | ### UniSpace generation | Benchmark | Breakdown | Scores | Overall ↑ | |---|---|---|---:| | GenEval | Single / Two / Count / Color / Position / Attribute | 0.98 / 0.92 / 0.69 / 0.88 / 0.83 / 0.73 | 0.84 | | DPG-Bench | Global / Entity / Attribute / Relation / Other | 84.80 / 92.26 / 90.00 / 94.97 / 88.80 | 86.49 | | OneIG EN | Align / Text / Reason / Style / Diversity | 0.860 / 0.937 / 0.311 / 0.467 / 0.233 | 0.561 | | OneIG ZH | Align / Text / Reason / Style / Diversity | 0.807 / 0.881 / 0.276 / 0.455 / 0.244 | 0.533 | ### UniSpace editing | Benchmark | Breakdown | Scores | Overall ↑ | |---|---|---|---:| | ImgEdit | Add / Adjust / Extract / Replace / Remove / Background / Style / Hybrid / Action | 4.53 / 4.38 / 3.61 / 4.67 / 4.42 / 4.23 / 4.55 / 2.70 / 4.47 | 4.28 | | GEdit EN | Semantic consistency / Perceptual quality | 8.287 / 7.055 | 7.407 | | GEdit ZH | Semantic consistency / Perceptual quality | 8.270 / 6.998 | 7.382 | Judge-based editing scores may vary with the evaluator endpoint and model version. The clean ImgEdit rerun generated all 737 expected outputs and scored 4.25 overall; category differences from the paper result were within 0.30. ## Intended use and limitations The release is intended for research on visual representation learning, multimodal inference, image generation, image editing, and reproducible benchmarking. Generated or edited images may contain factual, compositional, text-rendering, or perceptual errors. Users should independently assess outputs before using them in consequential settings. The release does not add application-specific safety guarantees beyond those of its base models. ## Citation ```bibtex @article{yan2026unispace, title = {UniSpace: Unified Visual Representation and Scalable Multimodal Modeling}, author = {Yan, Jinbo and Qiao, Limeng and Qin, Jie and He, Jun-Yan and Wu, Feize and Wan, Guanglu}, journal = {arXiv preprint arXiv:2608.08676}, year = {2026} } ``` ## License and third-party components Repository code is distributed under the licenses included in the GitHub release. Base-model and benchmark components retain their respective licenses and terms. See [`THIRD_PARTY.md`](https://github.com/yjb6/UniSpace/blob/main/THIRD_PARTY.md) before redistribution or commercial use.