--- license: llama3.1 base_model: meta-llama/Llama-3.1-8B-Instruct tags: - biomedical - genomics - variant-interpretation - lora - clinical-genomics - bioinformatics - research library_name: transformers pipeline_tag: text-generation --- # Model Card for KAU-BioMedLLM KAU-BioMedLLM is a research prototype for source-grounded biomedical variant interpretation. The current public release contains the LoRA adapter and documentation for a guarded report-generation system built around biomedical evidence retrieval, label-free variant scoring, citation enforcement, and abstention when evidence is insufficient. This repository contains two adapter releases: - **Root adapter:** Llama-3.1-8B-Instruct v0.2 LoRA adapter, recommended baseline for report generation. - **`qwen_v0_1/` adapter:** Qwen2.5-1.5B-Instruct v0.1 LoRA adapter, lightweight prototype/demo fallback. This repository does **not** redistribute full Llama or Qwen base model weights. Users must separately load the relevant base model according to its license terms. ## Table of Contents - [Model Details](#model-details) - [Repository Layout](#repository-layout) - [Model Description](#model-description) - [System Overview](#system-overview) - [Uses](#uses) - [Out-of-Scope Use](#out-of-scope-use) - [Training and Development Data](#training-and-development-data) - [Training Procedure](#training-procedure) - [Evaluation](#evaluation) - [Example Guarded Report](#example-guarded-report) - [Limitations](#limitations) - [How to Load the Adapter](#how-to-load-the-adapter) - [Compute Infrastructure](#compute-infrastructure) - [Ethical and Safety Considerations](#ethical-and-safety-considerations) - [Citation and Acknowledgement](#citation-and-acknowledgement) ## Model Details - **Model name:** KAU-BioMedLLM - **Repository:** `Babajaan/KAU-BioMedLLM` - **Developed by:** Babajan B, King Abdulaziz University research environment - **Model type:** Biomedical instruction-tuned LoRA adapter for report generation - **Primary base model:** `meta-llama/Llama-3.1-8B-Instruct` - **Prototype baseline:** `Qwen/Qwen2.5-1.5B-Instruct` - **Language:** English - **Domain:** Biomedical variant interpretation, clinical genomics research, bioinformatics evidence synthesis - **Release type:** Research-use adapter release - **Clinical status:** Not clinically validated; not for diagnosis, treatment, or patient-management decisions ## Repository Layout | Path | Adapter | Base model | Intended role | |---|---|---|---| | repository root | Llama v0.2 LoRA | `meta-llama/Llama-3.1-8B-Instruct` | Recommended report-generation baseline | | `qwen_v0_1/` | Qwen v0.1 LoRA | `Qwen/Qwen2.5-1.5B-Instruct` | Lightweight prototype and demo fallback | The root adapter is kept as the primary Hugging Face adapter so standard PEFT loading with `Babajaan/KAU-BioMedLLM` loads the Llama v0.2 adapter. The Qwen adapter is included in the same repository under `qwen_v0_1/` to avoid maintaining a second model repo. ## Model Description KAU-BioMedLLM was developed as part of a broader biomedical AI project to generate structured, source-grounded variant interpretation reports. The system combines two deliberately separated components: 1. **Independent score model:** a label-free machine learning model for variant scoring and temporal benchmarking. 2. **LLM report generator:** a LoRA-adapted instruction model for structured biomedical explanation. The central design choice is to avoid treating LLM-generated prose as a calibrated pathogenicity score. The numeric score and the narrative interpretation are separated, and the report-generation layer is guarded by schema validation, citation checks, and abstention rules. ## System Overview The broader KAU-BioMedLLM workflow uses: - ClinVar current and archived snapshots for variant labels and temporal validation. - PubMed and biomedical literature evidence for gene and phenotype context. - UniProt for protein/gene functional evidence. - AlphaFold structure information for residue-level structural context where mapping is reliable. - Ensembl/VEP-style transcript and protein residue mapping. - A guarded report pipeline that enforces JSON structure, citation support, and abstention when evidence is missing or ambiguous. The uploaded model artifact is the LoRA adapter used for the report-generation component. The repository also includes data provenance notes, evaluation summaries, manuscript tables, and inference instructions. ## Uses ### Intended Use KAU-BioMedLLM is intended for research and development in: - biomedical variant interpretation workflows; - genomics and bioinformatics report prototyping; - source-grounded biomedical evidence synthesis; - evaluating guarded LLM reporting pipelines; - studying leakage-aware score model design and temporal validation. ### Direct Use The adapter can be loaded with `meta-llama/Llama-3.1-8B-Instruct` using PEFT. It can generate structured research reports when provided with variant and evidence context. ### Downstream Use The model may be used as a starting point for research on: - citation-grounded biomedical report generation; - variant prioritization interfaces; - guarded medical LLM output; - separation of statistical score models from narrative explanation; - biomedical uncertainty and abstention behavior. ## Out-of-Scope Use KAU-BioMedLLM must not be used as: - a clinical diagnostic system; - a substitute for expert variant curation; - a medical advice generator; - a system for patient-management or treatment decisions; - a standalone pathogenicity classifier for clinical reporting; - a validated replacement for ACMG/AMP interpretation workflows. ## Training and Development Data The project used biomedical and genomics resources including: - ClinVar current and archived variant snapshots; - PubMed/literature-derived evidence examples; - UniProt protein/gene evidence; - AlphaFold structural evidence for selected proteins; - Ensembl/VEP-style transcript and residue mapping outputs; - internally generated structured instruction examples for report generation. The score model was intentionally designed around label-free features and excludes direct ClinVar significance text, review-star strength, gene identity, and external predictor scores such as REVEL, AlphaMissense, and CADD as training inputs. See `reports/DATA_PROVENANCE.md` for project-level data provenance and known version-recording limitations. ## Training Procedure The report-generation model was trained as a LoRA adapter rather than as a full base-model retraining run. The project developed two model stages: - **Qwen2.5-1.5B-Instruct v0.1:** lightweight prototype used to validate the basic structured-output and guarded-report pipeline. - **Llama-3.1-8B-Instruct v0.2:** stronger report-generation baseline selected for improved biomedical reasoning capacity and structured report generation. The LoRA approach keeps the released artifact compact and avoids redistributing the full base model weights. ## Evaluation The project separates score model evaluation from LLM report-generation evaluation. ### Broad First-Appearance Temporal Score Model The broad score model was evaluated using a first-appearance ClinVar temporal split: - Training snapshot: ClinVar 2023-12-26 - Test snapshot: ClinVar 2025-12-21 - Test set: labeled variants appearing in the 2025 snapshot and absent from the 2023 snapshot Reported benchmark: | Metric | Value | |---|---:| | Test variants | 509,974 | | AUROC | 0.99125 | | AUPRC | 0.96938 | | Brier score | 0.02903 | | ECE | 0.03428 | These metrics apply to the broad score model, not to the LLM text generator alone. ### Per-Consequence Breakdown The broad AUROC is not sufficient by itself because consequence classes differ strongly in class balance and difficulty. The table below reports the first-appearance temporal score model by consequence class. It is included to make clear that the headline broad score is partly consequence-class-driven. | Consequence | N | Pos | Neg | AUROC | AUPRC | Brier | ECE | |---|---:|---:|---:|---:|---:|---:|---:| | non-coding_transcript_variant | 34,607 | 257 | 34,350 | 0.91551 | 0.61517 | 0.00398 | 0.00219 | | 3_prime_UTR_variant | 3,861 | 24 | 3,837 | 0.89588 | 0.13295 | 0.00623 | 0.00464 | | other low-N | 534 | 336 | 198 | 0.87042 | 0.90320 | 0.13473 | 0.05293 | | unknown | 1,503 | 542 | 961 | 0.86056 | 0.75580 | 0.14478 | 0.05336 | | missense_variant | 67,847 | 12,943 | 54,904 | 0.81292 | 0.55438 | 0.18069 | 0.23174 | | splice_donor_variant | 8,438 | 8,238 | 200 | 0.81240 | 0.99205 | 0.01565 | 0.00245 | | 5_prime_UTR_variant | 6,609 | 68 | 6,541 | 0.77156 | 0.23015 | 0.00891 | 0.00739 | | inframe_deletion | 386 | 219 | 167 | 0.76963 | 0.80047 | 0.20247 | 0.08748 | | initiator_codon_variant | 713 | 615 | 98 | 0.74360 | 0.93131 | 0.09672 | 0.04770 | | intron_variant | 153,601 | 951 | 152,650 | 0.73241 | 0.03627 | 0.00619 | 0.00863 | | frameshift_variant | 38,296 | 37,947 | 349 | 0.72617 | 0.99567 | 0.00786 | 0.00094 | | nonsense | 22,375 | 22,034 | 341 | 0.70115 | 0.99182 | 0.01238 | 0.00779 | | splice_acceptor_variant | 7,209 | 7,081 | 128 | 0.70025 | 0.98999 | 0.01468 | 0.00421 | | synonymous_variant | 163,995 | 121 | 163,874 | 0.64395 | 0.00175 | 0.00074 | 0.00078 | Missense variants are materially harder than the broad all-variant benchmark. This is why the project reports missense-specific limitations separately and does not claim state-of-the-art missense prediction. ### Missense-Specific Limitation Missense-only prediction remained substantially harder: | Model / Predictor | Missense AUROC | |---|---:| | score_model v0 | 0.78497 | | missense v1 protein-feature score_model | 0.83644 | | AlphaMissense same-row comparison | 0.97214 | | REVEL same-row comparison | 0.97832 | This limitation is important: KAU-BioMedLLM should not be claimed as state-of-the-art for missense pathogenicity prediction. ### Report Guard Evaluation The guarded report layer was evaluated on a small report set: - Citation guard pass: 5/5 reports - Citation guard fail: 0/5 reports - Structure mapping available: 1/5 reports - Structure mapping abstained: 4/5 reports The abstention behavior is intentional: when evidence is missing, ambiguous, or not reliably mapped, the system should state uncertainty rather than fabricate confidence. ## Example Guarded Report The example below illustrates the intended report style: the statistical score-model field is separated from the LLM narrative, biomedical claims include source identifiers, and missing evidence triggers abstention rather than unsupported interpretation. ```json { "gene": "MC4R", "variant": "chr18:60371172:C>T", "final_interpretation": "Uncertain significance", "score_model_prediction": { "status": "abstain", "reason": "variant_not_in_existing_score_prediction_tables", "message": "No precomputed score_model prediction was available for this report input. Full arbitrary-variant scoring is deferred to v2.", "not_for_clinical_diagnosis": true }, "llm_interpretation": { "source": "KAU-BioMedLLM report generator", "reasoning_summary": "The variant is represented as a missense variation in MC4R. The available source label is uncertain significance. This does not establish clinical causality without transcript-level validation, allele frequency, segregation, phenotype match, functional data, and ACMG/AMP evidence review.", "limitations": "Research record only. Not a clinical diagnosis." }, "evidence_examples": { "gene_function": "MC4R functions in the leptin-melanocortin pathway and regulates energy homeostasis.", "source_ids": ["UniProt:P32245", "PMID:12646665", "PMID:32327598", "PMID:33858992"], "structure_context": "Use only when transcript, protein accession, and residue mapping can be reconciled." }, "guard_status": { "citation_check": "pass", "abstention_allowed": true, "clinical_use": "not permitted" } } ``` Full guarded report examples and summaries are included in the repository under `reports/full_reports/`. ## Limitations - The current evidence panel is limited and is not a genome-wide production knowledge base. - Missense prediction remains below specialist predictors such as REVEL and AlphaMissense. - Broad score model performance is partly driven by easier consequence classes. - Structure evidence is only used when transcript/protein/residue mapping can be reconciled. - Exact historical versions for some external resources require final manual verification before manuscript submission. - The model has not been prospectively validated in clinical settings. - Generated text can still contain errors and must be reviewed by domain experts. - The uploaded artifact is an adapter, not a standalone full model. ## How to Load the Adapters ### Llama v0.2 Adapter ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch base_model = "meta-llama/Llama-3.1-8B-Instruct" adapter_model = "Babajaan/KAU-BioMedLLM" tokenizer = AutoTokenizer.from_pretrained(base_model) model = AutoModelForCausalLM.from_pretrained( base_model, torch_dtype=torch.float16, device_map="auto", ) model = PeftModel.from_pretrained(model, adapter_model) model.eval() messages = [ { "role": "user", "content": ( "For research use only, generate a structured biomedical variant " "interpretation report for BRCA1 c.5096G>A p.Arg1699Gln. " "Include uncertainty and do not provide clinical advice." ), } ] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=700, temperature=0.2, do_sample=True, ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ### Qwen v0.1 Adapter The Qwen adapter is stored in the same repository under `qwen_v0_1/`. Use it for lightweight demos when Llama-3.1-8B hardware is unavailable. ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch base_model = "Qwen/Qwen2.5-1.5B-Instruct" adapter_model = "Babajaan/KAU-BioMedLLM" adapter_subfolder = "qwen_v0_1" tokenizer = AutoTokenizer.from_pretrained(base_model) model = AutoModelForCausalLM.from_pretrained( base_model, torch_dtype=torch.float16, device_map="auto", ) model = PeftModel.from_pretrained( model, adapter_model, subfolder=adapter_subfolder, ) model.eval() ``` ## Compute Infrastructure The project was developed and evaluated using the King Abdulaziz University Aziz High Performance Computing environment. The workflow used the Aziz HPC software environment, PBS job scheduling, and GPU resources including A100 GPU jobs where available. KAU Aziz HPC support resources: - Aziz Supercomputer Support Center: https://hpcc-kau.com/support/ - PBS and GPU job support documentation are available through the Aziz HPC knowledge base. ## Ethical and Safety Considerations Biomedical LLM systems can produce fluent but incorrect explanations. KAU-BioMedLLM therefore uses a guard-first design: - numeric score and LLM narrative are separated; - citations are required for biomedical claims; - reports can abstain when evidence is insufficient; - outputs are research-use only; - no clinical decision should be made from model output. Users should treat all outputs as draft research interpretations requiring expert review. ## Citation and Acknowledgement If you use this model or the associated code/reports, please cite the forthcoming project manuscript or repository when available. Manuscript status: in preparation. A formal citation will be added after manuscript submission or publication. Suggested acknowledgement: > This work used computational resources from the King Abdulaziz University Aziz High Performance Computing environment. ## License and Base Model Notes This repository contains LoRA adapters and project documentation. It does not redistribute full Llama or Qwen base model weights. Use of the root adapter with `meta-llama/Llama-3.1-8B-Instruct` is subject to the Meta Llama 3.1 Community License and Acceptable Use Policy. Use of the `qwen_v0_1/` adapter with `Qwen/Qwen2.5-1.5B-Instruct` is subject to Qwen license terms. Users are responsible for verifying license compatibility for their own use case.