Expand KAU-BioMedLLM model card and HPC acknowledgement

README.md

@@ -8,65 +8,270 @@ tags:
   - genomics
   - variant-interpretation
   - lora
+  - clinical-genomics
+  - bioinformatics
   - research
 library_name: transformers
 pipeline_tag: text-generation
 ---
 
-# KAUBioMED-LLM
+# Model Card for KAU-BioMedLLM
 
-KAUBioMED-LLM is a research prototype for guarded biomedical variant interpretation plus leakage-aware temporal variant scoring.
+KAU-BioMedLLM is a research prototype for source-grounded biomedical variant interpretation. The current public release contains the LoRA adapter and documentation for a guarded report-generation system built around biomedical evidence retrieval, label-free variant scoring, citation enforcement, and abstention when evidence is insufficient.
 
-## Components
+This repository does **not** redistribute the full Meta Llama-3.1-8B-Instruct base model. Users must separately have access to the base model according to the Meta Llama 3.1 license terms.
 
-- Qwen2.5-1.5B-Instruct v0.1: lightweight prototype report generator.
-- Llama-3.1-8B-Instruct v0.2: stronger guarded report-generation baseline.
-- score_model: scikit-learn HistGradientBoostingClassifier with isotonic calibration, using label-free broad variant features.
+## Table of Contents
 
-## Intended Use
+- [Model Details](#model-details)
+- [Model Description](#model-description)
+- [System Overview](#system-overview)
+- [Uses](#uses)
+- [Out-of-Scope Use](#out-of-scope-use)
+- [Training and Development Data](#training-and-development-data)
+- [Training Procedure](#training-procedure)
+- [Evaluation](#evaluation)
+- [Limitations](#limitations)
+- [How to Load the Adapter](#how-to-load-the-adapter)
+- [Compute Infrastructure](#compute-infrastructure)
+- [Ethical and Safety Considerations](#ethical-and-safety-considerations)
+- [Citation and Acknowledgement](#citation-and-acknowledgement)
 
-Research use only. Not for clinical diagnosis, treatment decisions, or patient management.
+## Model Details
 
-## Data and Provenance
+- **Model name:** KAU-BioMedLLM
+- **Repository:** `Babajaan/KAU-BioMedLLM`
+- **Developed by:** Babajan B, King Abdulaziz University research environment
+- **Model type:** Biomedical instruction-tuned LoRA adapter for report generation
+- **Primary base model:** `meta-llama/Llama-3.1-8B-Instruct`
+- **Prototype baseline:** `Qwen/Qwen2.5-1.5B-Instruct`
+- **Language:** English
+- **Domain:** Biomedical variant interpretation, clinical genomics research, bioinformatics evidence synthesis
+- **Release type:** Research-use adapter release
+- **Clinical status:** Not clinically validated; not for diagnosis, treatment, or patient-management decisions
 
-See `reports/DATA_PROVENANCE.md`.
+## Model Description
 
-## Key Results
+KAU-BioMedLLM was developed as part of a broader biomedical AI project to generate structured, source-grounded variant interpretation reports. The system combines two deliberately separated components:
 
-First-appearance temporal broad score_model test:
+1. **Independent score model:** a label-free machine learning model for variant scoring and temporal benchmarking.
+2. **LLM report generator:** a LoRA-adapted instruction model for structured biomedical explanation.
 
-- AUROC: 0.99125
-- AUPRC: 0.96938
-- Brier: 0.02903
-- ECE 10-bin: 0.03428
+The central design choice is to avoid treating LLM-generated prose as a calibrated pathogenicity score. The numeric score and the narrative interpretation are separated, and the report-generation layer is guarded by schema validation, citation checks, and abstention rules.
 
-Missense-only limitation:
+## System Overview
 
-- score_model v0 missense AUROC: 0.78497
-- missense v1 protein-feature AUROC: 0.83644
-- AlphaMissense same-row AUROC: 0.97214
-- REVEL same-row AUROC: 0.97832
+The broader KAU-BioMedLLM workflow uses:
 
-## Known Limitations
+- ClinVar current and archived snapshots for variant labels and temporal validation.
+- PubMed and biomedical literature evidence for gene and phenotype context.
+- UniProt for protein/gene functional evidence.
+- AlphaFold structure information for residue-level structural context where mapping is reliable.
+- Ensembl/VEP-style transcript and protein residue mapping.
+- A guarded report pipeline that enforces JSON structure, citation support, and abstention when evidence is missing or ambiguous.
 
-- Broad AUROC is driven partly by easy consequence classes; see `reports/score_model/per_consequence_breakdown.md`.
-- Missense performance remains below REVEL and AlphaMissense.
-- Evidence panel is limited to 20 genes and is not genome-wide.
-- Calibration is measured on a first-appearance ClinVar temporal split, not prospective clinical data.
-- Not validated for clinical deployment.
+The uploaded model artifact is the LoRA adapter used for the report-generation component. The repository also includes data provenance notes, evaluation summaries, manuscript tables, and inference instructions.
 
-## Base Model Attribution and License Notes
+## Uses
 
-Llama-derived artifacts are subject to the Meta Llama 3.1 Community License and Acceptable Use Policy. Qwen-derived prototype artifacts are subject to Qwen2.5 license terms. Verify license terms before public release.
+### Intended Use
 
-## Reproducibility Commands
+KAU-BioMedLLM is intended for research and development in:
 
-```bash
-cd /ddn/data/generic/bbabajan/KAUBioMED_LLM
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/build_label_free_features.py
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/train_label_free_score_model.py
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/evaluate_first_appearance_temporal.py
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/compare_external_predictors_first_appearance.py
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/kaubiomed_full_report.py
-/ddn/data/generic/bbabajan/envs/kaubiomed-llm/bin/python pipeline/apply_report_guards.py
+- biomedical variant interpretation workflows;
+- genomics and bioinformatics report prototyping;
+- source-grounded biomedical evidence synthesis;
+- evaluating guarded LLM reporting pipelines;
+- studying leakage-aware score model design and temporal validation.
+
+### Direct Use
+
+The adapter can be loaded with `meta-llama/Llama-3.1-8B-Instruct` using PEFT. It can generate structured research reports when provided with variant and evidence context.
+
+### Downstream Use
+
+The model may be used as a starting point for research on:
+
+- citation-grounded biomedical report generation;
+- variant prioritization interfaces;
+- guarded medical LLM output;
+- separation of statistical score models from narrative explanation;
+- biomedical uncertainty and abstention behavior.
+
+## Out-of-Scope Use
+
+KAU-BioMedLLM must not be used as:
+
+- a clinical diagnostic system;
+- a substitute for expert variant curation;
+- a medical advice generator;
+- a system for patient-management or treatment decisions;
+- a standalone pathogenicity classifier for clinical reporting;
+- a validated replacement for ACMG/AMP interpretation workflows.
+
+## Training and Development Data
+
+The project used biomedical and genomics resources including:
+
+- ClinVar current and archived variant snapshots;
+- PubMed/literature-derived evidence examples;
+- UniProt protein/gene evidence;
+- AlphaFold structural evidence for selected proteins;
+- Ensembl/VEP-style transcript and residue mapping outputs;
+- internally generated structured instruction examples for report generation.
+
+The score model was intentionally designed around label-free features and excludes direct ClinVar significance text, review-star strength, gene identity, and external predictor scores such as REVEL, AlphaMissense, and CADD as training inputs.
+
+See `reports/DATA_PROVENANCE.md` for project-level data provenance and known version-recording limitations.
+
+## Training Procedure
+
+The report-generation model was trained as a LoRA adapter rather than as a full base-model retraining run.
+
+The project developed two model stages:
+
+- **Qwen2.5-1.5B-Instruct v0.1:** lightweight prototype used to validate the basic structured-output and guarded-report pipeline.
+- **Llama-3.1-8B-Instruct v0.2:** stronger report-generation baseline selected for improved biomedical reasoning capacity and structured report generation.
+
+The LoRA approach keeps the released artifact compact and avoids redistributing the full base model weights.
+
+## Evaluation
+
+The project separates score model evaluation from LLM report-generation evaluation.
+
+### Broad First-Appearance Temporal Score Model
+
+The broad score model was evaluated using a first-appearance ClinVar temporal split:
+
+- Training snapshot: ClinVar 2023-12-26
+- Test snapshot: ClinVar 2025-12-21
+- Test set: labeled variants appearing in the 2025 snapshot and absent from the 2023 snapshot
+
+Reported benchmark:
+
+| Metric | Value |
+|---|---:|
+| Test variants | 509,974 |
+| AUROC | 0.99125 |
+| AUPRC | 0.96938 |
+| Brier score | 0.02903 |
+| ECE | 0.03428 |
+
+These metrics apply to the broad score model, not to the LLM text generator alone.
+
+### Missense-Specific Limitation
+
+Missense-only prediction remained substantially harder:
+
+| Model / Predictor | Missense AUROC |
+|---|---:|
+| score_model v0 | 0.78497 |
+| missense v1 protein-feature score_model | 0.83644 |
+| AlphaMissense same-row comparison | 0.97214 |
+| REVEL same-row comparison | 0.97832 |
+
+This limitation is important: KAU-BioMedLLM should not be claimed as state-of-the-art for missense pathogenicity prediction.
+
+### Report Guard Evaluation
+
+The guarded report layer was evaluated on a small report set:
+
+- Citation guard pass: 5/5 reports
+- Citation guard fail: 0/5 reports
+- Structure mapping available: 1/5 reports
+- Structure mapping abstained: 4/5 reports
+
+The abstention behavior is intentional: when evidence is missing, ambiguous, or not reliably mapped, the system should state uncertainty rather than fabricate confidence.
+
+## Limitations
+
+- The current evidence panel is limited and is not a genome-wide production knowledge base.
+- Missense prediction remains below specialist predictors such as REVEL and AlphaMissense.
+- Broad score model performance is partly driven by easier consequence classes.
+- Structure evidence is only used when transcript/protein/residue mapping can be reconciled.
+- Exact historical versions for some external resources require final manual verification before manuscript submission.
+- The model has not been prospectively validated in clinical settings.
+- Generated text can still contain errors and must be reviewed by domain experts.
+- The uploaded artifact is an adapter, not a standalone full model.
+
+## How to Load the Adapter
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+import torch
+
+base_model = "meta-llama/Llama-3.1-8B-Instruct"
+adapter_model = "Babajaan/KAU-BioMedLLM"
+
+tokenizer = AutoTokenizer.from_pretrained(base_model)
+model = AutoModelForCausalLM.from_pretrained(
+    base_model,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(model, adapter_model)
+model.eval()
+
+messages = [
+    {
+        "role": "user",
+        "content": (
+            "For research use only, generate a structured biomedical variant "
+            "interpretation report for BRCA1 c.5096G>A p.Arg1699Gln. "
+            "Include uncertainty and do not provide clinical advice."
+        ),
+    }
+]
+
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=700,
+        temperature=0.2,
+        do_sample=True,
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+
+## Compute Infrastructure
+
+The project was developed and evaluated using the King Abdulaziz University Aziz High Performance Computing environment. The workflow used the Aziz HPC software environment, PBS job scheduling, and GPU resources including A100 GPU jobs where available.
+
+KAU Aziz HPC support resources:
+
+- Aziz Supercomputer Support Center: https://hpcc-kau.com/support/
+- PBS and GPU job support documentation are available through the Aziz HPC knowledge base.
+
+## Ethical and Safety Considerations
+
+Biomedical LLM systems can produce fluent but incorrect explanations. KAU-BioMedLLM therefore uses a guard-first design:
+
+- numeric score and LLM narrative are separated;
+- citations are required for biomedical claims;
+- reports can abstain when evidence is insufficient;
+- outputs are research-use only;
+- no clinical decision should be made from model output.
+
+Users should treat all outputs as draft research interpretations requiring expert review.
+
+## Citation and Acknowledgement
+
+If you use this model or the associated code/reports, please cite the forthcoming project manuscript or repository when available.
+
+Suggested acknowledgement:
+
+> This work used computational resources from the King Abdulaziz University Aziz High Performance Computing environment.
+
+## License and Base Model Notes
+
+This repository contains a LoRA adapter and project documentation. It does not redistribute the full Llama base model. Use of the adapter with `meta-llama/Llama-3.1-8B-Instruct` is subject to the Meta Llama 3.1 Community License and Acceptable Use Policy. Qwen-derived prototype artifacts are subject to Qwen license terms. Users are responsible for verifying license compatibility for their own use case.
+