# CSMorgan-MEDVQA — Task 1 Fix-Pack

**Competition:** ImageCLEFmed-MEDVQA-GI-2026, Task 1 (Clinically Relevant VQA)
**Team:** CSMorgan-MEDVQA · Morgan State University, Computer Vision & AI Lab
**HF handle:** `sageofai`
**HF repo:** `sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1`
**Base model:** `Qwen/Qwen2.5-VL-7B-Instruct` + QLoRA adapter fine-tuned on Kvasir-VQA-x1

---

## Why this fix-pack exists

Our public leaderboard scores collapsed to **BLEU = 0.0**, **ROUGE-1 = 0.22**, **METEOR = 0.17**, while top systems sit at BLEU ≈ 0.38–0.46 and ROUGE-1 ≈ 0.84–0.89.

Diagnosing the previous `submission_task1.py` revealed the cause: the model was emitting full conversational sentences such as *"The image shows a polyp in the sigmoid colon."* while the Kvasir-VQA-x1 references are 1–3 word labels such as *"polyp"*. BLEU is a geometric mean of n-gram precisions, so a single verbose token kills every n ≥ 2 and the score floors at zero. ROUGE / METEOR partially survive (recall-based), which matches the exact pattern we saw.

The fix is a three-pronged pipeline that turns the existing VLM into a short-label generator without retraining:

1. **Strict label-only prompting.** A system message forbids hedging / "the image shows" / markdown, and the user prompt is routed by question type (yes/no, count, color, location, finding, diagnosis, instrument, anatomy, severity, other).
2. **Aggressive answer normalization.** ~60 VLM prefixes stripped, subordinate clauses cut, ~120 surface forms collapsed to canonical medical labels (`polyps → polyp`, `ulceration → ulcer`, `blood → bleeding`, `no abnormality → normal`, `erythematous mucosa → erythema`, `barretts esophagus → barrett's esophagus`, …), question-type-aware label extraction, and length control (compress > 6 words, reject > 12 words).
3. **Training answer-bank retrieval fallback.** Empty or off-script outputs fall back to the nearest valid training answer (Jaccard + SequenceMatcher + log-frequency prior, restricted to the matching question-type bucket).

Decoding is also switched from sampling (T = 0.1, top_k = 20, top_p = 0.7, max_tokens = 64) to **greedy** (T = 0.0, top_k = 1, max_tokens = 32), since exact-match metrics punish sampling variance.

---

## Repo layout

```
sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1/
├── submission_task1.py      # the only entrypoint medvqa will run
├── normalization.py         # imported as a sibling module
├── answer_bank.json         # produced locally by build_answer_bank.py, committed to the repo
├── requirements.txt
├── README.md                # this file
└── (adapter weights — already there)
```

The QLoRA adapter weights are already in the repo; this fix-pack only adds / replaces the 5 small files above. **`answer_bank.json` must be built locally and pushed alongside the script** — the submission container has no network access to rebuild it.

---

## Roll-out (run these locally, then push)

### 1. Build the answer bank from the Kvasir-VQA-x1 train split

```bash
pip install -U datasets
python build_answer_bank.py
```

This writes `answer_bank.json` (~a few MB) containing:

- `answer_frequencies` — global frequency of every observed training answer
- `by_question_type` — frequencies bucketed by yes_no / finding / count / color / …
- `by_question_class` — frequencies bucketed by the dataset's `question_class` field
- `by_question_text` — per-exact-question ranked answers (strongest signal when test questions repeat)
- `all_unique_answers` — flat set used for nearest-neighbour fallback

You'll see the top 30 most frequent answers printed as a sanity check. They should look short and clinical (`polyp`, `no`, `yes`, `normal`, `bleeding`, `ulcer`, …).

### 2. Sanity-check the normalization layer

```bash
python test_normalization.py
```

Expected output: `34/34 passed.` Covers yes/no edge cases, verbose model output, markdown wrapping, count words, color phrasing, canonical-map word boundaries, and category-filtered label extraction.

### 3. Dry-run the full inference script (optional but recommended)

```bash
python submission_task1.py
```

This loads the model + adapter, runs the validation split, writes `predictions_1.json`, and prints inline diagnostics. **Watch for the FLAGS section at the bottom.** It will warn if:

- more than 2 % of answers are empty
- average answer length > 4 words
- more than 10 % of answers are longer than 6 words
- any answers still contain "the image", "appears", "likely", "because", or markdown

If any flag fires, stop and inspect before submitting.

### 4. Independent diagnostic pass (no GPU needed)

```bash
python local_validate.py predictions_1.json
```

Exits 0 if clean, 1 if flagged, 2 if the file is malformed. Useful if you want to inspect a `predictions_1.json` produced elsewhere (e.g. by the validation container) without re-running inference.

### 5. Push and submit

Push the five files (and the rebuilt `answer_bank.json`) to the HF repo root, then:

```bash
pip install -U medvqa
medvqa validate --competition=gi-2026 --task=1 --repo_id=sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1
medvqa validate_and_submit --competition=gi-2026 --task=1 --repo_id=sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1
```

Deadline: **22 May 2026**.

---

## What's preserved from the original script

The "DO NOT EDIT" boundary in `submission_task1.py` is untouched:

- `SUBMISSION_INFO` dict (team name, affiliations, contact) is unchanged
- `HF_REPO_ID` is unchanged
- The `evaluate` block (bleu / rouge / meteor) is unchanged
- The output schema of `predictions_1.json` is unchanged (`index`, `img_id`, `question`, `answer`)
- The `medvqa validate_and_submit` instructions printed at the end are unchanged
- The PtEngine + BitsAndBytesConfig 4-bit nf4 model load is unchanged (same adapter, same base model)

All edits live inside the two `EDIT SECTION` blocks the template explicitly allows.

---

## File-by-file summary

| File | Purpose | Lines |
|---|---|---|
| `submission_task1.py` | Drop-in replacement for the broken script. Greedy decoding, system + routed-user prompt, calls `normalize_task1_answer` on every output, prints inline diagnostics. | ~290 |
| `normalization.py` | Core fix. Public API: `normalize_task1_answer`, `get_question_type`, `nearest_answer`, `build_answer_bank`. Self-contained, no heavy deps. | ~500 |
| `build_answer_bank.py` | Loads `SimulaMet/Kvasir-VQA-x1` train split, calls `build_answer_bank`, writes `answer_bank.json`. | ~50 |
| `test_normalization.py` | 34 torture cases covering every question type and every prefix / markdown / clause failure mode found during development. | ~120 |
| `local_validate.py` | Standalone diagnostic over an existing `predictions_1.json`. Same FLAGS logic as the inline diagnostics. | ~90 |
| `requirements.txt` | Pinned to the Kvasir-VQA-x1 reference setup (`ms-swift==3.8.0`, `qwen_vl_utils==0.0.11`). | — |

---

## Ablation hooks (if you have time before the deadline)

`submission_task1.py` is structured so you can run four modes by toggling two flags near the top:

| Mode | `USE_NORMALIZATION` | `USE_ANSWER_BANK` |
|---|---|---|
| raw_model | `False` | `False` |
| normalized_model | `True` | `False` |
| **normalized_with_answer_bank** (default) | `True` | `True` |
| answer_bank_only_baseline | — | run separately via `nearest_answer` only |

Submit the strongest mode. Empirically `normalized_with_answer_bank` should win, since the bank rescues the ~5–15 % of outputs that escape canonicalization.

---

## Contact

Peter Ojonugwa Ejiga — `ojeji1@morgan.edu`