README.md

metadata

title: GemmaSight
emoji: 🔬
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: false

GemmaSight 🔬 — Multimodal Colorectal Cancer Pathology Assistant

GemmaSight is a state-of-the-art multimodal AI pathology assistant designed to analyze H&E-stained colorectal cancer tissue patches to rapidly classify MSI-High (Microsatellite Instability-High) vs. MSS (Microsatellite Stable) status, bypassing the typical 2-4 week turnaround of traditional genomic sequencing.

---

🚀 Key Features (Five-Phase Pipeline)

1. Dual-Encoder Fusion: Features google/path-foundation (384-dimensional) and google/medsiglip-448 (1152-dimensional) frozen vision models, concatenated into a high-fidelity 1536-dimensional clinical embedding.
2. Classification MLP: A highly regularized, 3-layer dense neural network trained to optimize AUROC and output the final binary phenotype status and probability score.
3. FAISS Retrieval: Normalized cosine-similarity indexing (faiss.IndexFlatL2) to locate and retrieve the Top-3 matching reference cases from the clinical training registry for evidence-based decisions.
4. Spatial Explainability: Sliding-window occlusion (40x40 stride=20) computing exact confidence drop deltas smoothed with a Gaussian kernel to isolate and highlight the Top 20% most influential diagnostic regions.
5. Report Generation (MedGemma): Automatically prompts google/medgemma-1.5-4b-it in bfloat16 to fuse score, visual overlays, and historical matches into a structured clinical pathology brief under 200 words.

---

🛠️ Project File Structure

gemmasight/
├── requirements.txt           # Dependency requirements
├── config.py                 # Hyperparameters, paths, and mode configs
├── models/
│   ├── __init__.py
│   ├── feature_extractor.py  # Dual-encoder feature extractor (gated fallback)
│   ├── classifier.py         # 3-layer classification MLP PyTorch module
│   ├── retriever.py          # FAISS Flat index case retriever
│   └── report_generator.py   # MedGemma clinical report generator (simulator fallback)
├── utils/
│   ├── __init__.py
│   ├── occlusion.py          # Sliding-window occlusion visualizer
│   ├── preprocess.py         # PIL preprocessing transformations
│   └── visualization.py      # Premium custom Gradio layouts & CSS
├── train.py                  # MLP classifier training pipeline
├── inference.py              # End-to-end diagnosis flow orchestration
├── app.py                    # Entrypoint for the Gradio Blocks dashboard
└── README.md                 # Deployment & setup documentation

---

📦 Setup & Local Installation

Prerequisites

• Python 3.9+
• CUDA-enabled GPU (Highly recommended for live model weights, though CPU simulation mode is supported)

Step 1: Clone and install dependencies

git clone https://github.com/msaleem-aisci/gemmasight.git
cd gemmasight
pip install -r requirements.txt

Step 2: Configure Environment Variables & Gated Access

Ensure your HF_TOKEN has read permissions and your Hugging Face account has accepted the license agreements for:

• google/path-foundation
• google/medsiglip-448
• google/medgemma-1.5-4b-it

Export token:

export HF_TOKEN="your_huggingface_write_token_here"

If you are running in resource-constrained environments (e.g. standard CPUs or free spaces), ensure FORCE_SIMULATION = True is set in config.py to activate the high-fidelity simulator fallbacks.

---

🔬 Running Training & Inference

1. Optimize Classifier (MLP)

Run training on pre-extracted embeddings. If no embeddings are found in data/, the pipeline will automatically generate a randomized synthetic dataset baseline to facilitate quick local testing:

python train.py

This builds the FAISS case indices and saves the trained classification model weights into weights/best_classifier.pt.

2. Launch Gradio Clinical Dashboard

Launch the dashboard on localhost (defaults to port 7860):

python app.py

---

🚀 Deployment to Hugging Face Spaces

1. Create a new Space on Hugging Face (Choose the Gradio SDK, preferably selecting a GPU hardware tier like T4 Small for fast inference).
2. Set the HF_TOKEN as a Secret under your Hugging Face Space settings.
3. Push your repository to the Hugging Face Space Git remote:

git remote add hf https://huggingface.co/spaces/YOUR_USER_OR_ORG/YOUR_SPACE_NAME
git push -u hf main

4. The space will automatically install dependencies from requirements.txt and launch app.py.