---
title: GemmaSight
emoji: 🔬
colorFrom: green
colorTo: blue

sdk: gradio
sdk_version: 4.44.0

app_file: app.py
pinned: false
---

# GemmaSight 🔬 — Multimodal Colorectal Cancer Pathology Assistant


GemmaSight is a state-of-the-art multimodal AI pathology assistant designed to analyze H&E-stained colorectal cancer tissue patches to rapidly classify MSI-High (Microsatellite Instability-High) vs. MSS (Microsatellite Stable) status, bypassing the typical 2-4 week turnaround of traditional genomic sequencing.

---

## 🚀 Key Features (Five-Phase Pipeline)
1. **Dual-Encoder Fusion**: Features `google/path-foundation` (384-dimensional) and `google/medsiglip-448` (1152-dimensional) frozen vision models, concatenated into a high-fidelity 1536-dimensional clinical embedding.
2. **Classification MLP**: A highly regularized, 3-layer dense neural network trained to optimize AUROC and output the final binary phenotype status and probability score.
3. **FAISS Retrieval**: Normalized cosine-similarity indexing (`faiss.IndexFlatL2`) to locate and retrieve the Top-3 matching reference cases from the clinical training registry for evidence-based decisions.
4. **Spatial Explainability**: Sliding-window occlusion (40x40 stride=20) computing exact confidence drop deltas smoothed with a Gaussian kernel to isolate and highlight the **Top 20%** most influential diagnostic regions.
5. **Report Generation (MedGemma)**: Automatically prompts `google/medgemma-1.5-4b-it` in bfloat16 to fuse score, visual overlays, and historical matches into a structured clinical pathology brief under 200 words.

---

## 🛠️ Project File Structure

```
gemmasight/
├── requirements.txt           # Dependency requirements
├── config.py                 # Hyperparameters, paths, and mode configs
├── models/
│   ├── __init__.py
│   ├── feature_extractor.py  # Dual-encoder feature extractor (gated fallback)
│   ├── classifier.py         # 3-layer classification MLP PyTorch module
│   ├── retriever.py          # FAISS Flat index case retriever
│   └── report_generator.py   # MedGemma clinical report generator (simulator fallback)
├── utils/
│   ├── __init__.py
│   ├── occlusion.py          # Sliding-window occlusion visualizer
│   ├── preprocess.py         # PIL preprocessing transformations
│   └── visualization.py      # Premium custom Gradio layouts & CSS
├── train.py                  # MLP classifier training pipeline
├── inference.py              # End-to-end diagnosis flow orchestration
├── app.py                    # Entrypoint for the Gradio Blocks dashboard
└── README.md                 # Deployment & setup documentation
```

---

## 📦 Setup & Local Installation

### Prerequisites
- Python 3.9+
- CUDA-enabled GPU (Highly recommended for live model weights, though CPU simulation mode is supported)

### Step 1: Clone and install dependencies
```bash
git clone https://github.com/msaleem-aisci/gemmasight.git
cd gemmasight
pip install -r requirements.txt
```

### Step 2: Configure Environment Variables & Gated Access
Ensure your `HF_TOKEN` has read permissions and your Hugging Face account has accepted the license agreements for:
- [google/path-foundation](https://huggingface.co/google/path-foundation)
- [google/medsiglip-448](https://huggingface.co/google/medsiglip-448)
- [google/medgemma-1.5-4b-it](https://huggingface.co/google/medgemma-1.5-4b-it)

Export token:
```bash
export HF_TOKEN="your_huggingface_write_token_here"
```

If you are running in resource-constrained environments (e.g. standard CPUs or free spaces), ensure `FORCE_SIMULATION = True` is set in `config.py` to activate the high-fidelity simulator fallbacks.

---

## 🔬 Running Training & Inference

### 1. Optimize Classifier (MLP)
Run training on pre-extracted embeddings. If no embeddings are found in `data/`, the pipeline will automatically generate a randomized synthetic dataset baseline to facilitate quick local testing:
```bash
python train.py
```
This builds the FAISS case indices and saves the trained classification model weights into `weights/best_classifier.pt`.

### 2. Launch Gradio Clinical Dashboard
Launch the dashboard on localhost (defaults to port `7860`):
```bash
python app.py
```

---

## 🚀 Deployment to Hugging Face Spaces

1. Create a new Space on Hugging Face (Choose the **Gradio** SDK, preferably selecting a GPU hardware tier like T4 Small for fast inference).
2. Set the `HF_TOKEN` as a Secret under your Hugging Face Space settings.
3. Push your repository to the Hugging Face Space Git remote:
```bash
git remote add hf https://huggingface.co/spaces/YOUR_USER_OR_ORG/YOUR_SPACE_NAME
git push -u hf main
```
4. The space will automatically install dependencies from `requirements.txt` and launch `app.py`.