File size: 4,966 Bytes

e5f5711

---
license: cc-by-nc-4.0
library_name: transformers
tags:
- mobile-o
- multimodal
- unified-model
- vision-language
- text-to-image
- image-understanding
- on-device
- mobile
pipeline_tag: text-to-image
datasets:
- Amshaker/Mobile-O-Post-Train
- Amshaker/Mobile-O-SFT
- Amshaker/Mobile-O-Pre-Train
base_model:
- Efficient-Large-Model/Sana_600M_512px_diffusers
- apple/FastVLM-0.5B
---

<div align="center">

<h1>
  <img src="https://github.com/Amshaker/Mobile-O/blob/main/assets/mobile-o-logo.png?raw=true" width="30" /> Mobile-O-0.5B
</h1>

**Unified Multimodal Understanding and Generation on Mobile Device**

<p>
<a href="https://arxiv.org/abs/2602.20161"><img src="https://img.shields.io/badge/arXiv-2602.20161-b31b1b.svg" alt="arXiv"></a>
<a href="https://github.com/Amshaker/Mobile-O"><img src="https://img.shields.io/badge/GitHub-Code-black.svg" alt="Code"></a>
<a href="https://amshaker.github.io/Mobile-O/"><img src="https://img.shields.io/badge/🌐-Project_Page-2563eb.svg" alt="Project Page"></a>
<a href="https://mobileo.cvmbzuai.com/"><img src="https://img.shields.io/badge/🚀-Live_Demo-10b981.svg" alt="Demo"></a>
<a href="https://huggingface.co/collections/Amshaker/mobile-o-datasets"><img src="https://img.shields.io/badge/🤗-Datasets-yellow.svg" alt="Datasets"></a>
<a href="https://apps.apple.com/app/mobile-o/id6759238106"><img src="https://img.shields.io/badge/-App_Store-black.svg" alt="App Store"></a>
</p>

</div>

## 📌 Overview

Mobile-O-0.5B is a compact unified vision–language–diffusion model that performs both **multimodal understanding** (VQA, OCR, reasoning) and **image generation** within a single architecture, designed for mobile and edge deployment.

| Spec | Detail |
|------|--------|
| **Total Parameters** | 1.6B |
| **Image Resolution** | 512×512 |
| **Image Generation** | ~3 seconds on iPhone |
| **Visual Understanding** | ~0.4 seconds on iPhone |
| **Memory Footprint** | < 2GB |

## 🎯 Supported Tasks

| Task | Input → Output |
|------|---------------|
| 💬 Conversational AI | Text → Text |
| 👁️ Image Understanding | Image + Text → Text |
| 🖼️ Image Generation | Text → Image |
| ✏️ Image Editing | Image + Text → Image |

## 🚀 Quick Start

### Download

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Amshaker/Mobile-O-0.5B",
    repo_type="model",
    local_dir="checkpoints",
    allow_patterns=["final_merged_model_23620/*"]
)
```

### Image Understanding

```bash
python infer_und.py \
    --model_path checkpoints/final_merged_model_23620/ \
    --image_path assets/cute_cat.png \
    --prompt "What is in the image?"
```

### Image Generation

```bash
python infer_gen.py \
    --model_path checkpoints/final_merged_model_23620/ \
    --prompt "A vibrant tropical rainforest scene with a scarlet macaw perched on a moss-covered branch"
```

### Image Editing

```bash
python infer_edit.py \
    --model_path checkpoints/final_merged_model_23620/ \
    --image_path assets/cute_cat.png \
    --prompt "Make the cat wear a hat"
```

## 🏗️ Architecture

Mobile-O consists of three main components:

- **Vision-Language Model (VLM):** [FastVLM-0.5B](https://github.com/apple/ml-fastvlm) — FastViT vision encoder + Qwen2-0.5B language backbone
- **Diffusion Decoder:** [SANA-600M-512](https://github.com/NVlabs/Sana) — lightweight linear DiT with VAE for 512×512 generation
- **Mobile Conditioning Projector (MCP):** ~2.4M param connector using layerwise feature fusion with temperature-scaled weights, depthwise-separable 1D convolutions, and efficient channel attention

## 🏋️ Training

Trained in three stages:

1. **Pre-training** — Cross-modal alignment on [4M text-image pairs](https://huggingface.co/datasets/Amshaker/Mobile-O-Pre-Train)
2. **SFT** — Supervised fine-tuning on [~105K curated pairs](https://huggingface.co/datasets/Amshaker/Mobile-O-SFT)
3. **Post-training** — Unified multimodal training on [~105K quadruplets](https://huggingface.co/datasets/Amshaker/Mobile-O-Post-Train)

## 🔗 Related Resources

| Resource | Link |
|----------|------|
| 🤗 Mobile-O-1.5B | [Model](https://huggingface.co/Amshaker/Mobile-O-1.5B) |
| 🤗 Mobile-O-0.5B-iOS | [iOS Components](https://huggingface.co/Amshaker/Mobile-O-0.5B-iOS) |
| 📱 iOS App Source Code | [Mobile-O-App](https://github.com/Amshaker/Mobile-O/tree/main/Mobile-O-App) |

## 📄 Citation

```bibtex
@article{shaker2026mobileo,
  title={Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device},
  author={Shaker, Abdelrahman and Heakl, Ahmed and Muhammad, Jaseel and Thawkar, Ritesh and Thawakar, Omkar and Li, Senmao and Cholakkal, Hisham and Reid, Ian and Xing, Eric P. and Khan, Salman and Khan, Fahad Shahbaz},
  journal={arXiv preprint arXiv:2602.20161},
  year={2026}
}
```

## ⚖️ License

Released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). For research purposes only.