--- license: mit license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE language: - en tags: - mlx - apple-silicon - vision-language-model - multimodal - phi-4 - quantized - 4bit - siglip - document-understanding - chart-understanding - ocr pipeline_tag: image-text-to-text library_name: mlx base_model: microsoft/Phi-4-multimodal-instruct datasets: - lmms-lab/DocVQA - lmms-lab/ai2d - MMMU/MMMU - HuggingFaceM4/ChartQA - lmms-lab/textvqa - echo840/OCRBench - derek-thomas/ScienceQA - AI4Math/MathVista model-index: - name: Phi-4-multimodal-instruct-mlx-4bit results: - task: type: image-text-to-text dataset: name: AI2D type: lmms-lab/ai2d split: test metrics: - type: accuracy value: 83.0 name: Accuracy (n=100) - task: type: image-text-to-text dataset: name: ChartQA type: HuggingFaceM4/ChartQA split: test metrics: - type: relaxed_accuracy value: 86.0 name: Relaxed Accuracy (n=100) - task: type: image-text-to-text dataset: name: DocVQA type: lmms-lab/DocVQA split: validation metrics: - type: anls value: 82.8 name: ANLS (n=100) - task: type: image-text-to-text dataset: name: TextVQA type: lmms-lab/textvqa split: validation metrics: - type: accuracy value: 80.0 name: Accuracy (n=100) - task: type: image-text-to-text dataset: name: OCRBench type: echo840/OCRBench split: test metrics: - type: score value: 840 name: Score/1000 (n=100) - task: type: image-text-to-text dataset: name: ScienceQA type: derek-thomas/ScienceQA split: test metrics: - type: accuracy value: 95.8 name: Accuracy (n=48, image-only) - task: type: image-text-to-text dataset: name: MathVista type: AI4Math/MathVista split: testmini metrics: - type: accuracy value: 58.0 name: Accuracy (n=100) --- # Phi-4-Multimodal-Instruct — MLX 4-bit A 4-bit quantized [Apple MLX](https://github.com/ml-explore/mlx) conversion of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) for native inference on Apple Silicon. **Converted by [Ferox AI](https://ferox.ca)** · Vision-language inference on MacBook / Mac Studio / Mac Pro without cloud dependencies. | | | |---|---| | **Parameters** | 5.6B (pre-LoRA-fusion) | | **Quantization** | 4-bit, group_size=64 (backbone only; SigLIP encoder remains FP16) | | **Disk size** | ~3.9 GB | | **Base model** | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) | | **License** | MIT | | **Modality** | Vision + Text (Phase 1; audio deferred) | > **Other variants:** [bf16 (full precision)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-bf16) · [8-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit) ## Quickstart ```python from mlx_vlm import load, generate model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-4bit") output = generate( model, processor, "Describe this image in detail.", ["path/to/image.jpg"], max_tokens=512, verbose=False, ) print(output) ``` Requires `mlx-vlm >= 0.1.0` with Phi-4-MM architecture support. Install dependencies: ```bash pip install mlx-vlm>=0.1.0 mlx>=0.22.0 ``` ## Benchmark Results Evaluated with our internal evaluation harness on a single Apple Silicon device. Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16 — direct comparison should account for both the precision difference and sample-size variance. | Benchmark | This Model (4-bit) | bf16 | Microsoft FP16 (full dataset) | Metric | |-----------|:------------------:|:----:|:-----------------------------:|--------| | **AI2D** | 83.0 | 90.0 | 82.3 | Accuracy | | **ChartQA** | **86.0** | 85.0 | 81.4 | Relaxed Accuracy | | **DocVQA** | 82.8 | 86.2 | 93.2 | ANLS | | **MathVista** | 58.0 | 58.0 | 62.4 | Accuracy | | **MMMU** | 24.0 | 31.0 | 55.1 | Accuracy | | **OCRBench** | 840 | 840 | 844 | Score / 1000 | | **ScienceQA** | 95.8† | 100.0† | 97.5 | Accuracy | | **TextVQA** | **80.0** | 82.0 | 75.6 | Accuracy | † ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded). ### Quantization impact Across all benchmarks, 4-bit quantization produces a mean accuracy delta of −2.2 percentage points relative to bf16 — within the expected range for 4-bit group quantization on a model of this scale. ### Note on MMMU The 100-sample MMMU scores (24.0% 4-bit, 31.0% bf16) fall well below Microsoft's reported 55.1%. To isolate the cause, we ran a **full 900-sample MMMU validation** on the lossless bf16 variant and obtained **27.9%** — consistent with the subset, which confirms the gap is **not** caused by quantization or weight conversion. We were unable to reproduce Microsoft's 55.1% and attribute the difference to evaluation-harness and answer-extraction handling for MMMU's multiple-choice format (prompt formatting and option parsing), rather than to the model's underlying capability — which is better reflected by the document-, chart-, OCR-, and science-focused benchmarks above. ## Architecture | Component | Details | |-----------|---------| | **Backbone** | Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE positional encoding (131K context) | | **Vision encoder** | SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152 | | **Vision projection** | 2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features | | **Vision LoRA** | rank=256, alpha=512 (~370M parameters) — **pre-fused** into backbone weights before quantization | | **Image preprocessing** | Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference | | **Quantization** | 4-bit with group_size=64. Applied to backbone linear layers only; SigLIP encoder weights remain in FP16 | ### Weight provenance Weights are converted from `microsoft/Phi-4-multimodal-instruct` using a deterministic pipeline: 1. Download source checkpoint (PyTorch safetensors) 2. Fuse vision LoRA adapters into backbone weights (eliminates runtime adapter overhead) 3. Remap weight keys to MLX naming conventions 4. Transpose LoRA matrices (PEFT → MLX format) 5. Quantize backbone to 4-bit (SigLIP excluded) 6. Serialize as MLX safetensors The conversion and quantization pipeline is deterministic and fully reproducible from the base model. ## Intended Use This model is designed for **local, on-device vision-language inference** on Apple Silicon hardware. Suitable applications include: - Document understanding and extraction (invoices, forms, reports) - Chart and diagram interpretation - Visual question answering - OCR and text recognition in images - Educational content analysis ### Out of scope - Audio processing (Phase 2, not included in this release) - Production deployment without application-level safety filtering - Use cases requiring guaranteed factual accuracy without human verification ## Limitations - **100-sample evaluations.** Benchmark scores are computed on subsets, not full datasets. Expect variance relative to full-dataset evaluations. - **Vision-only.** This is a Phase 1 release covering the vision modality. Audio support from the original Phi-4-multimodal architecture is not included. - **No runtime LoRA switching.** Vision LoRA adapters are pre-fused; the model cannot dynamically swap adapters. - **Apple Silicon required.** MLX is designed for Apple's unified memory architecture (M1/M2/M3/M4). This model will not run on CUDA or CPU-only systems. ## Citation If you use this model in your work, please cite: ```bibtex @misc{feroxai2026phi4mlx, title={Phi-4-Multimodal-Instruct MLX Conversion}, author={Ferox AI}, year={2026}, url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit}, note={4-bit quantized MLX port of microsoft/Phi-4-multimodal-instruct} } ``` ## Acknowledgments - **Microsoft Research** for the [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) model and technical report - **Apple MLX team** for the [MLX framework](https://github.com/ml-explore/mlx) - **Prince Canuma** for [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)