Model card for MIPHEI-ViT

⚠️ This Hugging Face repository contains documentation and metadata only.
No model weights or source code are hosted in this repository.
The source code and usage instructions are available in the official GitHub repository, while the model weights are distributed in the v1.0.0 GitHub release.

---

MIPHEI-ViT is a deep learning model that predicts 16-channel multiplex immunofluorescence (mIF) images from standard H&E-stained histology images. It uses a U-Net-style architecture with a ViT foundation model (H-Optimus-0) as the encoder, inspired by the ViTMatte model.

This work is described in our paper:

“MIPHEI-vit: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models.”

Please see the publication for full results and details.

The model was trained on a processed version of the ORION-CRC dataset, available here: 🔗 MIPHEI-ViT Dataset on Zenodo

It takes H&E image tiles as input and outputs 16-channel mIF predictions for the following markers: Hoechst, CD31, CD45, CD68, CD4, FOXP3, CD8a, CD45RO, CD20, PD-L1, CD3e, CD163, E-cadherin, Ki67, Pan-CK, SMA

For optimal performances, input H&E images should come from colon tissue and be scanned at 0.5 µm/pixel (20x magnification). However, because the model is built on a large ViT foundation (H-Optimus-0), so you may try applying it to other tissue type as well.

Figure: Overview of the MIPHEI-ViT architecture.

This model was developed as part of research funded by Sanofi and ANRT.