---
title: Neuron Quantification using AI — MedCLIPSeg
emoji: 🧠
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit
---

# Neuron Quantification using AI — MedCLIPSeg variant

Iman Sabir Ezzat, Randa K Ismail, Ayden Chavez, Marisa Zallocchi, PhD, Steven Fernandes, PhD

A variant of the neuron tracer that applies the **MedCLIPSeg** approach
(Koleilat et al., *Probabilistic Vision–Language Adaptation for Data-Efficient
and Generalizable Medical Image Segmentation*, CVPR 2026): a **text prompt**
describes the target ("nerve fibers …") and a frozen **CLIP** backbone produces a
**fiber-probability map**, which is then traced into a skeleton.

## What the app does
- **Neurofilament (MIP):** the maximum-intensity projection of the input channel.
- **Tracked skeleton (white on black):** the MedCLIP fiber-probability map is
  thresholded, skeletonised and pruned into a fiber skeleton.
- **Total trace length (µm):** the spacing-aware total length of that MedCLIP
  skeleton, reported above the images. The text prompt and probability threshold
  are editable.

## Few-shot training (included)
A runnable, few-shot version of the MedCLIPSeg approach — **frozen CLIP encoders +
a text-conditioned decoder** — is trained here on **8 animals** (1A, 1B, 2A, 3A,
3B, 4Aa, 4Bb, 5A) and **tested on the unseen C1 and C2b**:

```bash
python train_medclip_fewshot.py     # runs on CPU/MPS; saves weights_medclip_decoder.pt
```

**Result: held-out (C1 + C2b) mean Dice ≈ 0.53** (per image 0.36–0.60; train Dice
= 1.0, i.e. memorised). The segmentation lands on the fiber band and is far
better than zero-shot, but it is **coarse** — CLIP features are 14×14, so thin
fibers are only roughly captured, and the label is IMARIS's approximate
reconstruction. The app loads `weights_medclip_decoder.pt` automatically and uses
the trained model.

## Honest scope
The full authors' model (`reference_medclipseg/`, PVL adapters + probabilistic
attention) needs a **CUDA GPU**; the included few-shot decoder is the part that
runs here. Either way the MedCLIP output is a **segmentation prior** — the
**trustworthy quantification (length, fibers, diameter) comes from the classical
tracer**, and the *length number* does not generalize reliably across animals
(measured earlier). Without a trained decoder the app falls back to zero-shot
CLIP, which is only a coarse prior.

## To get the real, trained MedCLIPSeg
The original authors' code is bundled in **`reference_medclipseg/`** (needs a
**CUDA GPU**). Two steps:

1. **Build the dataset** (no manual masks needed — uses IMARIS's reconstruction
   as the label):
   ```bash
   python prepare_medclip_data.py
   # -> data/Neurofilament/{Train,Val,Test}_Folder/{img,label} + Prompts_Folder/*.xlsx
   ```
2. **Fine-tune** the model on a GPU:
   ```bash
   cd reference_medclipseg
   pip install -r requirements.txt
   python train.py --config-file configs/Neurofilament.yaml
   python test.py  --config-file configs/Neurofilament.yaml   # segmentation + uncertainty
   ```
   (`configs/Neurofilament.yaml` uses a BiomedCLIP backbone; switch `CLIP_MODEL`
   to `clip`/`pubmedclip`/`unimedclip` as desired.)

## Honest limitations (unchanged from the main project)
- The training **label** is IMARIS's *reconstructed* skeleton (from its segment
  coordinates), which is approximate, and IMARIS itself **over-traces** the true
  fibre length (~3–4×). So a trained model would imitate IMARIS, not ground truth.
- Predicting IMARIS's numbers does **not** generalize well across animals
  (measured R² ≈ 0 in earlier experiments). MedCLIPSeg's segmentation may look
  cleaner, but the **length number** remains unreliable on unseen animals.
- For exact IMARIS values, read the IMARIS statistics directly (as in the other
  tools).

## Files
- `app.py` — the Gradio variant (classical trace + MedCLIP maps).
- `medclipseg.py` — runnable text-prompted CLIP segmentation + uncertainty.
- `prepare_medclip_data.py` — builds the MedCLIPSeg dataset from CZIs + IMARIS
  reconstruction masks.
- `reference_medclipseg/` — the original MedCLIPSeg code (for GPU fine-tuning) +
  `configs/Neurofilament.yaml`.
- `processing.py` — the shared image-processing / tracing pipeline.