---
license: other
license_name: sanofi-non-commercial
license_link: LICENSE
language:
- en
- fr
base_model:
- bioptimus/H-optimus-0
library_name: timm
tags:
- histopathology
- image-to-image
- vit
- cell-classification
- image-translation
pipeline_tag: image-to-image
---
# Model card for MIPHEI-ViT

<p align="center">
  <img src="logo.svg" alt="MIPHEI-ViT Logo" style="height:250px;">
</p>

<p align="center">
  <a href="https://www.sciencedirect.com/science/article/pii/S0010482526001277" target="_blank" rel="noopener noreferrer">
    <img src="https://img.shields.io/badge/CBM-Paper-orange?logo=elsevier" height="25">
  </a>
  <a href="https://github.com/Sanofi-Public/MIPHEI-ViT" target="_blank" rel="noopener noreferrer">
    <img src="https://img.shields.io/badge/GitHub-Code-black?logo=github" height="25">
  </a>
  <a href="https://zenodo.org/records/15340874" target="_blank" rel="noopener noreferrer">
    <img src="https://img.shields.io/badge/Zenodo-Dataset-blue?logo=zenodo" height="25">
  </a>
  <a href="https://colab.research.google.com/github/Sanofi-Public/MIPHEI-ViT/blob/main/notebooks/colab_inference.ipynb">
    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="25">
  </a>
  <a href="https://miphei-vit-demo.minesparis.psl.eu/" target="_blank" rel="noopener noreferrer">
    <img src="https://img.shields.io/badge/Gradio-Demo-ffb000?logo=gradio" alt="Gradio Demo" height="25">
  </a>
</p>


> <span style="color:red"><strong>⚠️ This Hugging Face repository contains documentation and metadata only.</strong></span>
> <br>
> <span style="color:red">No model weights or source code are hosted in this repository.
> <br>
> The source code and usage instructions are available in the official <a href="https://github.com/Sanofi-Public/MIPHEI-ViT">GitHub repository</a>, while the model weights are distributed in the <a href="https://github.com/Sanofi-Public/MIPHEI-ViT/releases/tag/v1.0.0">v1.0.0 GitHub release</a>.</span>

---

**MIPHEI-ViT** is a deep learning model that predicts 16-channel **multiplex immunofluorescence (mIF)** images from standard **H&E-stained histology images**. It uses a **U-Net-style architecture** with a **ViT foundation model (H-Optimus-0)** as the encoder, inspired by the ViTMatte model.

This work is described in our paper:

**“MIPHEI-vit: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models.”**

Please see the publication for full results and details.

The model was trained on a processed version of the ORION-CRC dataset, available here: [🔗 MIPHEI-ViT Dataset on Zenodo](https://zenodo.org/records/15340874)

It takes H&E image tiles as input and outputs **16-channel mIF predictions** for the following markers: **Hoechst, CD31, CD45, CD68, CD4, FOXP3, CD8a, CD45RO, CD20, PD-L1, CD3e, CD163, E-cadherin, Ki67, Pan-CK, SMA**

For optimal performances, input H&E images should come from **colon tissue** and be scanned at **0.5 µm/pixel (20x magnification)**. However, because the model is built on a large ViT foundation (H-Optimus-0), so you may try applying it to other tissue type as well.

<p align="center">
  <img src="https://github.com/Sanofi-Public/MIPHEI-ViT/blob/main/figures/figure_architecture.png?raw=true" alt="MIPHEI-ViT Architecture" style="height:300px;">
</p>
<p align="center">
  <em>Figure: Overview of the MIPHEI-ViT architecture.</em>
</p>

This model was developed as part of research funded by **Sanofi** and **ANRT**.