nsr51324/CortexRAG

Medical RAG System

A retrieval-augmented generation (RAG) system for answering medical questions using a curated English medical Question-Answer knowledge base.

The system combines semantic retrieval (Sentence Transformers + FAISS), medical query expansion, Cross-Encoder reranking, evidence deduplication, confidence gating, and LLM-based answer generation.

Important: This system is for research and educational purposes only. It is not a medical diagnostic system and should not replace professional medical advice.

---

1. Pipeline

User Question
      │
      ▼
Query Expansion            (lay terms → medical terms)
      │
      ▼
Sentence Transformer Embedding
      │
      ▼
FAISS Vector Search        (Top 20 candidates)
      │
      ▼
Cross-Encoder Reranking    (Top 6 evidence)
      │
      ▼
Near-Duplicate Removal
      │
      ▼
Confidence Gate  ──reject──▶  "Insufficient evidence"
      │ pass
      ▼
LLM Generation
      │
      ▼
Evidence-Based Answer (with doc_id citations)

2. Knowledge Base

~16,384 medical Question-Answer records with columns Question, Answer, Category, doc_id.

3. Retrieval

• Embedding model: sentence-transformers/all-MiniLM-L6-v2, embeddings normalized so FAISS IndexFlatIP behaves as cosine similarity.
• Query Expansion: a small lay-term → medical-term dictionary is appended to the query before embedding (e.g. underactive thyroid → hypothyroidism, high blood sugar → hyperglycemia). The original query is never replaced, only extended.
• Reranking: cross-encoder/ms-marco-MiniLM-L-6-v2 scores (query, Answer) — not (query, Question) — since many records share the same question with different answers; this identifies which evidence is actually useful.
• Evidence deduplication: near-identical answers are collapsed via difflib.SequenceMatcher (threshold 0.92).

4. Confidence Gate

Evidence is only sent to the LLM if all conditions hold:

MIN_RERANK_SCORE = -8
MIN_SIMILARITY_FLOOR = 0.40
MIN_SUPPORT_COUNT = 2
SUPPORT_SCORE = -5.0

1. At least one result exists.
2. Top rerank score ≥ MIN_RERANK_SCORE.
3. Top similarity ≥ MIN_SIMILARITY_FLOOR.
4. At least MIN_SUPPORT_COUNT results score ≥ SUPPORT_SCORE.

Otherwise, the system explicitly refuses to answer rather than guessing.

5. Generation

LLM: Groq-hosted openai/gpt-oss-20b, instructed to answer only from retrieved evidence, cite sources by [doc_id], flag disagreement between sources, avoid diagnosis/prescriptions, and include a medical disclaimer. API key is read from GROQ_API_KEY (never hard-coded).

6. Example

Q: What signs might suggest that my thyroid is not producing enough hormones? Evidence: 6 thyroid-related records retrieved and reranked. Answer: Evidence-based summary (fatigue, feeling cold, weight gain, constipation, menstrual changes, hair loss) + disclaimer.

Out-of-domain example — Q: What is the best treatment for a broken leg? Retrieval returned diabetic-foot documents (lexically similar), but the gate/LLM correctly identified them as inappropriate evidence and refused to answer — demonstrating that similarity alone doesn't guarantee relevance.

7. Evaluation Summary

Metric	Result
Knowledge Base Size	16,384 records
Self-Retrieval Top-1	100%
Retrieval Recall@1 / @5 / @10	69.0% / 89.9% / 95.9%
Retrieval MRR	1.000
Retrieval + Reranker Recall@1 / @5 / @10	8.3% / 22.2% / 38.8%
Retrieval + Reranker MRR	0.237
Category Recall (retrieval / +reranker)	97% / 98%
Confidence Gate Accuracy	88% (n=50)
Example End-to-End Latency	0.99 sec

Key finding: the general-purpose Cross-Encoder scores notably worse than plain retrieval on this duplicate-question-based benchmark. This isn't hidden — it indicates the reranker (trained for general passage relevance) doesn't align well with medical relevance judgments, and needs validation against a manually reviewed gold set before being trusted in production. A semi-automatic gold-set workflow (exact duplicates + semantic candidates ≥0.90 similarity, human-reviewed 1/0 labels) is included for this purpose; Category-match is used only as a secondary sanity check, not ground truth.

The confidence-gate threshold was tuned via 5-fold cross-validation rather than a single split, to check stability rather than overfit to one small sample.

8. Limitations

• Knowledge base doesn't cover every condition/scenario.
• Semantic similarity ≠ appropriate evidence (see broken-leg example).
• Reranker not fine-tuned on this medical domain yet.
• Manual gold set is still small — evaluation should be treated as preliminary.
• Not for diagnosis, emergencies, prescriptions, or personalized treatment.

9. Installation

pip install sentence-transformers faiss-cpu numpy pandas openpyxl groq

import os
GROQ_API_KEY = os.environ["GROQ_API_KEY"]  # never hard-code keys

10. Future Work

Complete manual gold-set annotation → re-evaluate reranker against it → consider a medical-domain reranker → expand confidence-gate validation set → add citations/monitoring/auth → deploy behind a REST API → evaluate faithfulness/hallucination separately from retrieval quality.

11. Responsible Use

Not a diagnostic tool, clinical decision-support system, prescription system, or emergency service. Consult a qualified healthcare professional for medical decisions.

References

• Sentence Transformers
• FAISS
• Cross-Encoder