README.md

metadata

title: Neuron Quantification using AI — MedCLIPSeg
emoji: 🧠
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit

Neuron Quantification using AI — MedCLIPSeg variant

Iman Sabir Ezzat, Randa K Ismail, Ayden Chavez, Marisa Zallocchi, PhD, Steven Fernandes, PhD

A variant of the neuron tracer that applies the MedCLIPSeg approach (Koleilat et al., Probabilistic Vision–Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation, CVPR 2026): a text prompt describes the target ("nerve fibers …") and a frozen CLIP backbone produces a fiber-probability map, which is then traced into a skeleton.

What the app does

• Neurofilament (MIP): the maximum-intensity projection of the input channel.
• Tracked skeleton (white on black): the MedCLIP fiber-probability map is thresholded, skeletonised and pruned into a fiber skeleton.
• Total trace length (µm): the spacing-aware total length of that MedCLIP skeleton, reported above the images. The text prompt and probability threshold are editable.

Few-shot training (included)

A runnable, few-shot version of the MedCLIPSeg approach — frozen CLIP encoders + a text-conditioned decoder — is trained here on 8 animals (1A, 1B, 2A, 3A, 3B, 4Aa, 4Bb, 5A) and tested on the unseen C1 and C2b:

python train_medclip_fewshot.py     # runs on CPU/MPS; saves weights_medclip_decoder.pt

Result: held-out (C1 + C2b) mean Dice ≈ 0.53 (per image 0.36–0.60; train Dice = 1.0, i.e. memorised). The segmentation lands on the fiber band and is far better than zero-shot, but it is coarse — CLIP features are 14×14, so thin fibers are only roughly captured, and the label is IMARIS's approximate reconstruction. The app loads weights_medclip_decoder.pt automatically and uses the trained model.

Honest scope

The full authors' model (reference_medclipseg/, PVL adapters + probabilistic attention) needs a CUDA GPU; the included few-shot decoder is the part that runs here. Either way the MedCLIP output is a segmentation prior — the trustworthy quantification (length, fibers, diameter) comes from the classical tracer, and the length number does not generalize reliably across animals (measured earlier). Without a trained decoder the app falls back to zero-shot CLIP, which is only a coarse prior.

To get the real, trained MedCLIPSeg

The original authors' code is bundled in reference_medclipseg/ (needs a CUDA GPU). Two steps:

1. Build the dataset (no manual masks needed — uses IMARIS's reconstruction as the label):

   python prepare_medclip_data.py
   # -> data/Neurofilament/{Train,Val,Test}_Folder/{img,label} + Prompts_Folder/*.xlsx
   

2. Fine-tune the model on a GPU:

   cd reference_medclipseg
   pip install -r requirements.txt
   python train.py --config-file configs/Neurofilament.yaml
   python test.py  --config-file configs/Neurofilament.yaml   # segmentation + uncertainty
   

   (configs/Neurofilament.yaml uses a BiomedCLIP backbone; switch CLIP_MODEL to clip/pubmedclip/unimedclip as desired.)

Honest limitations (unchanged from the main project)

• The training label is IMARIS's reconstructed skeleton (from its segment coordinates), which is approximate, and IMARIS itself over-traces the true fibre length (~3–4×). So a trained model would imitate IMARIS, not ground truth.
• Predicting IMARIS's numbers does not generalize well across animals (measured R² ≈ 0 in earlier experiments). MedCLIPSeg's segmentation may look cleaner, but the length number remains unreliable on unseen animals.
• For exact IMARIS values, read the IMARIS statistics directly (as in the other tools).

Files

• app.py — the Gradio variant (classical trace + MedCLIP maps).
• medclipseg.py — runnable text-prompted CLIP segmentation + uncertainty.
• prepare_medclip_data.py — builds the MedCLIPSeg dataset from CZIs + IMARIS reconstruction masks.
• reference_medclipseg/ — the original MedCLIPSeg code (for GPU fine-tuning) + configs/Neurofilament.yaml.
• processing.py — the shared image-processing / tracing pipeline.