MAGEN-O-MAKE (ViT-B/16)

Vision-language model for dermatology, pretrained with MAGEN (Multi-Agent data GENeration) and O-MAKE (Ontology-based Multi-Aspect Knowledge-Enhanced pretraining).

MAGEN synthesizes knowledge-enriched image descriptions through a foundation-model-assisted captioning and retrieval-based verification pipeline. O-MAKE decomposes those long clinical texts into distinct knowledge aspects and aligns them with image features at both the global and patch level, guided by a dermatological ontology.

• 📄 Paper: IEEE TMI (arXiv:2512.03445)
• 💻 Code: github.com/XiejiLi/MAGEN-O-MAKE

Model details

Architecture	CLIP ViT-B/16, 512-d embedding, 77-token text encoder
Parameters	149.6 M
Initialization	OpenAI CLIP ViT-B/16
Pretraining data	Derm1M-AgentAug — ~400K dermatological image–text pairs augmented by MAGEN
Schedule	15 epochs, batch size 2048, lr 1e-4, AdamW (wd 0.1), 1500 warmup steps
Image preprocessing	resize 224 (bicubic) → center crop 224 → OpenAI CLIP normalization

Usage

import torch
from PIL import Image
import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:Xieji-Li/MAGEN-O-MAKE')
tokenizer = open_clip.get_tokenizer('hf-hub:Xieji-Li/MAGEN-O-MAKE')
model.eval()

# The six PAD-UFES-20 classes.
labels = ['nevus', 'basal cell carcinoma', 'actinic keratosis',
          'seborrheic keratosis', 'squamous cell carcinoma', 'melanoma']

# A biopsy-confirmed melanoma from the PAD-UFES-20 test split.
image = preprocess(Image.open('PAT_611_1158_156.png').convert('RGB')).unsqueeze(0)
text = tokenizer([f'This is a skin image of {c}' for c in labels])

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)[0]

for label, p in sorted(zip(labels, probs.tolist()), key=lambda kv: -kv[1]):
    print(f'{label:25s} {p:.4f}')

melanoma                  0.9227
seborrheic keratosis      0.0682
nevus                     0.0063
basal cell carcinoma      0.0011
actinic keratosis         0.0010
squamous cell carcinoma   0.0006

The image is from PAD-UFES-20; the code repository links a bundle of all downstream datasets. This single-template prompt is the simplest possible use; the benchmarks below ensemble 8 templates (OPENAI_SKIN_TEMPLATES in the code repository).

Zero-shot results

Measured with this checkpoint using the evaluation pipeline in the code repository (bash script/zeroshot_eval.sh), 8-template prompt ensemble:

Benchmark	Classes	Metric	Score
PAD-UFES-20	6	AUROC / Accuracy	0.9176 / 0.6675
Fitzpatrick17K	113	Top-1 / Top-5	0.3716 / 0.6620
SNU	134	Top-1 / Top-5	0.3898 / 0.7235
SD-128	128	Top-1 / Top-5	0.4595 / 0.7711
Daffodil	5	Top-1	0.8321
SD-tails (SD-198 \ SD-128)	70	Top-1 / Top-5	0.5565 / 0.8301
SNU-tails (<15 samples/class)	85	Top-1 / Top-5	0.4573 / 0.7882

The last two rows are long-tail splits: rare conditions the model never saw a label for during pretraining.

Files

File	Description
open_clip_model.safetensors	Weights in open_clip format (recommended)
open_clip_pytorch_model.bin	Same weights, pickle format
open_clip_config.json	Model and preprocessing configuration
O-MAKE_epoch_15.pt	Full training checkpoint (weights + optimizer + scaler), for resuming pretraining or use with src/test.py --resume

Intended use and limitations

This model is a research artifact, released for non-commercial research use only (CC BY-NC-ND 4.0). It is not a medical device and must not be used for diagnosis, triage, or any other clinical decision-making.

Known limitations:

• Trained on web-sourced dermatological images; skin-tone, geographic, and condition coverage is uneven, and performance varies accordingly across subgroups.
• Zero-shot accuracy on fine-grained benchmarks with 100+ classes is well below clinical reliability.
• Predictions depend on the prompt wording and on the candidate label set supplied by the caller.
• Part of the pretraining captions are model-generated (MAGEN) and were verified by retrieval rather than by clinicians, so residual label noise is expected.

Citation

@article{li2025multi,
  title={Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation},
  author={Li, Xieji and Yan, Siyuan and Liu, Yingsheng and Soyer, H Peter and Janda, Monika and Mar, Victoria and Ge, Zongyuan},
  journal={arXiv preprint arXiv:2512.03445},
  year={2025}
}

The MICCAI'25 conference version this extends:

@misc{yan2025makemultiaspectknowledgeenhancedvisionlanguage,
      title={MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment}, 
      author={Siyuan Yan and Xieji Li and Ming Hu and Yiwen Jiang and Zhen Yu and Zongyuan Ge},
      year={2025},
      eprint={2505.09372},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.09372}, 
}