sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1

CSMorgan-MEDVQA — Task 1 Fix-Pack

Competition: ImageCLEFmed-MEDVQA-GI-2026, Task 1 (Clinically Relevant VQA) Team: CSMorgan-MEDVQA · Morgan State University, Computer Vision & AI Lab HF handle: sageofai HF repo: sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1 Base model: Qwen/Qwen2.5-VL-7B-Instruct + QLoRA adapter fine-tuned on Kvasir-VQA-x1

---

Why this fix-pack exists

Our public leaderboard scores collapsed to BLEU = 0.0, ROUGE-1 = 0.22, METEOR = 0.17, while top systems sit at BLEU ≈ 0.38–0.46 and ROUGE-1 ≈ 0.84–0.89.

Diagnosing the previous submission_task1.py revealed the cause: the model was emitting full conversational sentences such as "The image shows a polyp in the sigmoid colon." while the Kvasir-VQA-x1 references are 1–3 word labels such as "polyp". BLEU is a geometric mean of n-gram precisions, so a single verbose token kills every n ≥ 2 and the score floors at zero. ROUGE / METEOR partially survive (recall-based), which matches the exact pattern we saw.

The fix is a three-pronged pipeline that turns the existing VLM into a short-label generator without retraining:

1. Strict label-only prompting. A system message forbids hedging / "the image shows" / markdown, and the user prompt is routed by question type (yes/no, count, color, location, finding, diagnosis, instrument, anatomy, severity, other).
2. Aggressive answer normalization. ~60 VLM prefixes stripped, subordinate clauses cut, ~120 surface forms collapsed to canonical medical labels (polyps → polyp, ulceration → ulcer, blood → bleeding, no abnormality → normal, erythematous mucosa → erythema, barretts esophagus → barrett's esophagus, …), question-type-aware label extraction, and length control (compress > 6 words, reject > 12 words).
3. Training answer-bank retrieval fallback. Empty or off-script outputs fall back to the nearest valid training answer (Jaccard + SequenceMatcher + log-frequency prior, restricted to the matching question-type bucket).

Decoding is also switched from sampling (T = 0.1, top_k = 20, top_p = 0.7, max_tokens = 64) to greedy (T = 0.0, top_k = 1, max_tokens = 32), since exact-match metrics punish sampling variance.

---

Repo layout

sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1/
├── submission_task1.py      # the only entrypoint medvqa will run
├── normalization.py         # imported as a sibling module
├── answer_bank.json         # produced locally by build_answer_bank.py, committed to the repo
├── requirements.txt
├── README.md                # this file
└── (adapter weights — already there)

The QLoRA adapter weights are already in the repo; this fix-pack only adds / replaces the 5 small files above. answer_bank.json must be built locally and pushed alongside the script — the submission container has no network access to rebuild it.

---

Roll-out (run these locally, then push)

1. Build the answer bank from the Kvasir-VQA-x1 train split

pip install -U datasets
python build_answer_bank.py

This writes answer_bank.json (~a few MB) containing:

• answer_frequencies — global frequency of every observed training answer
• by_question_type — frequencies bucketed by yes_no / finding / count / color / …
• by_question_class — frequencies bucketed by the dataset's question_class field
• by_question_text — per-exact-question ranked answers (strongest signal when test questions repeat)
• all_unique_answers — flat set used for nearest-neighbour fallback

You'll see the top 30 most frequent answers printed as a sanity check. They should look short and clinical (polyp, no, yes, normal, bleeding, ulcer, …).

2. Sanity-check the normalization layer

python test_normalization.py

Expected output: 34/34 passed. Covers yes/no edge cases, verbose model output, markdown wrapping, count words, color phrasing, canonical-map word boundaries, and category-filtered label extraction.

3. Dry-run the full inference script (optional but recommended)

python submission_task1.py

This loads the model + adapter, runs the validation split, writes predictions_1.json, and prints inline diagnostics. Watch for the FLAGS section at the bottom. It will warn if:

• more than 2 % of answers are empty
• average answer length > 4 words
• more than 10 % of answers are longer than 6 words
• any answers still contain "the image", "appears", "likely", "because", or markdown

If any flag fires, stop and inspect before submitting.

4. Independent diagnostic pass (no GPU needed)

python local_validate.py predictions_1.json

Exits 0 if clean, 1 if flagged, 2 if the file is malformed. Useful if you want to inspect a predictions_1.json produced elsewhere (e.g. by the validation container) without re-running inference.

5. Push and submit

Push the five files (and the rebuilt answer_bank.json) to the HF repo root, then:

pip install -U medvqa
medvqa validate --competition=gi-2026 --task=1 --repo_id=sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1
medvqa validate_and_submit --competition=gi-2026 --task=1 --repo_id=sageofai/Qwen25VL-MEDVQA-GI-S1-subtask1

Deadline: 22 May 2026.

---

What's preserved from the original script

The "DO NOT EDIT" boundary in submission_task1.py is untouched:

• SUBMISSION_INFO dict (team name, affiliations, contact) is unchanged
• HF_REPO_ID is unchanged
• The evaluate block (bleu / rouge / meteor) is unchanged
• The output schema of predictions_1.json is unchanged (index, img_id, question, answer)
• The medvqa validate_and_submit instructions printed at the end are unchanged
• The PtEngine + BitsAndBytesConfig 4-bit nf4 model load is unchanged (same adapter, same base model)

All edits live inside the two EDIT SECTION blocks the template explicitly allows.

---

File-by-file summary

File	Purpose	Lines
submission_task1.py	Drop-in replacement for the broken script. Greedy decoding, system + routed-user prompt, calls normalize_task1_answer on every output, prints inline diagnostics.	~290
normalization.py	Core fix. Public API: normalize_task1_answer, get_question_type, nearest_answer, build_answer_bank. Self-contained, no heavy deps.	~500
build_answer_bank.py	Loads SimulaMet/Kvasir-VQA-x1 train split, calls build_answer_bank, writes answer_bank.json.	~50
test_normalization.py	34 torture cases covering every question type and every prefix / markdown / clause failure mode found during development.	~120
local_validate.py	Standalone diagnostic over an existing predictions_1.json. Same FLAGS logic as the inline diagnostics.	~90
requirements.txt	Pinned to the Kvasir-VQA-x1 reference setup (ms-swift==3.8.0, qwen_vl_utils==0.0.11).	—

---

Ablation hooks (if you have time before the deadline)

submission_task1.py is structured so you can run four modes by toggling two flags near the top:

Mode	USE_NORMALIZATION	USE_ANSWER_BANK
raw_model	False	False
normalized_model	True	False
normalized_with_answer_bank (default)	True	True
answer_bank_only_baseline	—	run separately via nearest_answer only

Submit the strongest mode. Empirically normalized_with_answer_bank should win, since the bank rescues the ~5–15 % of outputs that escape canonicalization.

---

Contact

Peter Ojonugwa Ejiga — ojeji1@morgan.edu