Replace root adapter with Llama-3.1-8B v0.2 LoRA adapter

README.md

@@ -1,335 +1,207 @@
 ---
-license: llama3.1
 base_model: meta-llama/Llama-3.1-8B-Instruct
-tags:
-  - biomedical
-  - genomics
-  - variant-interpretation
-  - lora
-  - clinical-genomics
-  - bioinformatics
-  - research
-library_name: transformers
+library_name: peft
 pipeline_tag: text-generation
+tags:
+- base_model:adapter:meta-llama/Llama-3.1-8B-Instruct
+- lora
+- transformers
 ---
 
-# Model Card for KAU-BioMedLLM
+# Model Card for Model ID
 
-KAU-BioMedLLM is a research prototype for source-grounded biomedical variant interpretation. The current public release contains the LoRA adapter and documentation for a guarded report-generation system built around biomedical evidence retrieval, label-free variant scoring, citation enforcement, and abstention when evidence is insufficient.
+<!-- Provide a quick summary of what the model is/does. -->
 
-This repository does **not** redistribute the full Meta Llama-3.1-8B-Instruct base model. Users must separately have access to the base model according to the Meta Llama 3.1 license terms.
 
-## Table of Contents
-
-- [Model Details](#model-details)
-- [Model Description](#model-description)
-- [System Overview](#system-overview)
-- [Uses](#uses)
-- [Out-of-Scope Use](#out-of-scope-use)
-- [Training and Development Data](#training-and-development-data)
-- [Training Procedure](#training-procedure)
-- [Evaluation](#evaluation)
-- [Example Guarded Report](#example-guarded-report)
-- [Limitations](#limitations)
-- [How to Load the Adapter](#how-to-load-the-adapter)
-- [Compute Infrastructure](#compute-infrastructure)
-- [Ethical and Safety Considerations](#ethical-and-safety-considerations)
-- [Citation and Acknowledgement](#citation-and-acknowledgement)
 
 ## Model Details
 
-- **Model name:** KAU-BioMedLLM
-- **Repository:** `Babajaan/KAU-BioMedLLM`
-- **Developed by:** Babajan B, King Abdulaziz University research environment
-- **Model type:** Biomedical instruction-tuned LoRA adapter for report generation
-- **Primary base model:** `meta-llama/Llama-3.1-8B-Instruct`
-- **Prototype baseline:** `Qwen/Qwen2.5-1.5B-Instruct`
-- **Language:** English
-- **Domain:** Biomedical variant interpretation, clinical genomics research, bioinformatics evidence synthesis
-- **Release type:** Research-use adapter release
-- **Clinical status:** Not clinically validated; not for diagnosis, treatment, or patient-management decisions
-
-## Model Description
+### Model Description
 
-KAU-BioMedLLM was developed as part of a broader biomedical AI project to generate structured, source-grounded variant interpretation reports. The system combines two deliberately separated components:
+<!-- Provide a longer summary of what this model is. -->
 
-1. **Independent score model:** a label-free machine learning model for variant scoring and temporal benchmarking.
-2. **LLM report generator:** a LoRA-adapted instruction model for structured biomedical explanation.
 
-The central design choice is to avoid treating LLM-generated prose as a calibrated pathogenicity score. The numeric score and the narrative interpretation are separated, and the report-generation layer is guarded by schema validation, citation checks, and abstention rules.
 
-## System Overview
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-The broader KAU-BioMedLLM workflow uses:
+### Model Sources [optional]
 
-- ClinVar current and archived snapshots for variant labels and temporal validation.
-- PubMed and biomedical literature evidence for gene and phenotype context.
-- UniProt for protein/gene functional evidence.
-- AlphaFold structure information for residue-level structural context where mapping is reliable.
-- Ensembl/VEP-style transcript and protein residue mapping.
-- A guarded report pipeline that enforces JSON structure, citation support, and abstention when evidence is missing or ambiguous.
+<!-- Provide the basic links for the model. -->
 
-The uploaded model artifact is the LoRA adapter used for the report-generation component. The repository also includes data provenance notes, evaluation summaries, manuscript tables, and inference instructions.
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
 ## Uses
 
-### Intended Use
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-KAU-BioMedLLM is intended for research and development in:
+### Direct Use
 
-- biomedical variant interpretation workflows;
-- genomics and bioinformatics report prototyping;
-- source-grounded biomedical evidence synthesis;
-- evaluating guarded LLM reporting pipelines;
-- studying leakage-aware score model design and temporal validation.
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-### Direct Use
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
 
-The adapter can be loaded with `meta-llama/Llama-3.1-8B-Instruct` using PEFT. It can generate structured research reports when provided with variant and evidence context.
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-### Downstream Use
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-The model may be used as a starting point for research on:
+## How to Get Started with the Model
 
-- citation-grounded biomedical report generation;
-- variant prioritization interfaces;
-- guarded medical LLM output;
-- separation of statistical score models from narrative explanation;
-- biomedical uncertainty and abstention behavior.
+Use the code below to get started with the model.
 
-## Out-of-Scope Use
+[More Information Needed]
 
-KAU-BioMedLLM must not be used as:
+## Training Details
 
-- a clinical diagnostic system;
-- a substitute for expert variant curation;
-- a medical advice generator;
-- a system for patient-management or treatment decisions;
-- a standalone pathogenicity classifier for clinical reporting;
-- a validated replacement for ACMG/AMP interpretation workflows.
+### Training Data
 
-## Training and Development Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-The project used biomedical and genomics resources including:
+[More Information Needed]
 
-- ClinVar current and archived variant snapshots;
-- PubMed/literature-derived evidence examples;
-- UniProt protein/gene evidence;
-- AlphaFold structural evidence for selected proteins;
-- Ensembl/VEP-style transcript and residue mapping outputs;
-- internally generated structured instruction examples for report generation.
+### Training Procedure
 
-The score model was intentionally designed around label-free features and excludes direct ClinVar significance text, review-star strength, gene identity, and external predictor scores such as REVEL, AlphaMissense, and CADD as training inputs.
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-See `reports/DATA_PROVENANCE.md` for project-level data provenance and known version-recording limitations.
+#### Preprocessing [optional]
 
-## Training Procedure
+[More Information Needed]
 
-The report-generation model was trained as a LoRA adapter rather than as a full base-model retraining run.
 
-The project developed two model stages:
+#### Training Hyperparameters
 
-- **Qwen2.5-1.5B-Instruct v0.1:** lightweight prototype used to validate the basic structured-output and guarded-report pipeline.
-- **Llama-3.1-8B-Instruct v0.2:** stronger report-generation baseline selected for improved biomedical reasoning capacity and structured report generation.
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-The LoRA approach keeps the released artifact compact and avoids redistributing the full base model weights.
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
 
 ## Evaluation
 
-The project separates score model evaluation from LLM report-generation evaluation.
-
-### Broad First-Appearance Temporal Score Model
-
-The broad score model was evaluated using a first-appearance ClinVar temporal split:
-
-- Training snapshot: ClinVar 2023-12-26
-- Test snapshot: ClinVar 2025-12-21
-- Test set: labeled variants appearing in the 2025 snapshot and absent from the 2023 snapshot
-
-Reported benchmark:
-
-| Metric | Value |
-|---|---:|
-| Test variants | 509,974 |
-| AUROC | 0.99125 |
-| AUPRC | 0.96938 |
-| Brier score | 0.02903 |
-| ECE | 0.03428 |
-
-These metrics apply to the broad score model, not to the LLM text generator alone.
-
-### Per-Consequence Breakdown
-
-The broad AUROC is not sufficient by itself because consequence classes differ strongly in class balance and difficulty. The table below reports the first-appearance temporal score model by consequence class. It is included to make clear that the headline broad score is partly consequence-class-driven.
-
-| Consequence | N | Pos | Neg | AUROC | AUPRC | Brier | ECE |
-|---|---:|---:|---:|---:|---:|---:|---:|
-| non-coding_transcript_variant | 34,607 | 257 | 34,350 | 0.91551 | 0.61517 | 0.00398 | 0.00219 |
-| 3_prime_UTR_variant | 3,861 | 24 | 3,837 | 0.89588 | 0.13295 | 0.00623 | 0.00464 |
-| other low-N | 534 | 336 | 198 | 0.87042 | 0.90320 | 0.13473 | 0.05293 |
-| unknown | 1,503 | 542 | 961 | 0.86056 | 0.75580 | 0.14478 | 0.05336 |
-| missense_variant | 67,847 | 12,943 | 54,904 | 0.81292 | 0.55438 | 0.18069 | 0.23174 |
-| splice_donor_variant | 8,438 | 8,238 | 200 | 0.81240 | 0.99205 | 0.01565 | 0.00245 |
-| 5_prime_UTR_variant | 6,609 | 68 | 6,541 | 0.77156 | 0.23015 | 0.00891 | 0.00739 |
-| inframe_deletion | 386 | 219 | 167 | 0.76963 | 0.80047 | 0.20247 | 0.08748 |
-| initiator_codon_variant | 713 | 615 | 98 | 0.74360 | 0.93131 | 0.09672 | 0.04770 |
-| intron_variant | 153,601 | 951 | 152,650 | 0.73241 | 0.03627 | 0.00619 | 0.00863 |
-| frameshift_variant | 38,296 | 37,947 | 349 | 0.72617 | 0.99567 | 0.00786 | 0.00094 |
-| nonsense | 22,375 | 22,034 | 341 | 0.70115 | 0.99182 | 0.01238 | 0.00779 |
-| splice_acceptor_variant | 7,209 | 7,081 | 128 | 0.70025 | 0.98999 | 0.01468 | 0.00421 |
-| synonymous_variant | 163,995 | 121 | 163,874 | 0.64395 | 0.00175 | 0.00074 | 0.00078 |
-
-Missense variants are materially harder than the broad all-variant benchmark. This is why the project reports missense-specific limitations separately and does not claim state-of-the-art missense prediction.
-
-### Missense-Specific Limitation
-
-Missense-only prediction remained substantially harder:
-
-| Model / Predictor | Missense AUROC |
-|---|---:|
-| score_model v0 | 0.78497 |
-| missense v1 protein-feature score_model | 0.83644 |
-| AlphaMissense same-row comparison | 0.97214 |
-| REVEL same-row comparison | 0.97832 |
-
-This limitation is important: KAU-BioMedLLM should not be claimed as state-of-the-art for missense pathogenicity prediction.
-
-### Report Guard Evaluation
-
-The guarded report layer was evaluated on a small report set:
-
-- Citation guard pass: 5/5 reports
-- Citation guard fail: 0/5 reports
-- Structure mapping available: 1/5 reports
-- Structure mapping abstained: 4/5 reports
-
-The abstention behavior is intentional: when evidence is missing, ambiguous, or not reliably mapped, the system should state uncertainty rather than fabricate confidence.
-
-## Example Guarded Report
-
-The example below illustrates the intended report style: the statistical score-model field is separated from the LLM narrative, biomedical claims include source identifiers, and missing evidence triggers abstention rather than unsupported interpretation.
-
-```json
-{
-  "gene": "MC4R",
-  "variant": "chr18:60371172:C>T",
-  "final_interpretation": "Uncertain significance",
-  "score_model_prediction": {
-    "status": "abstain",
-    "reason": "variant_not_in_existing_score_prediction_tables",
-    "message": "No precomputed score_model prediction was available for this report input. Full arbitrary-variant scoring is deferred to v2.",
-    "not_for_clinical_diagnosis": true
-  },
-  "llm_interpretation": {
-    "source": "KAU-BioMedLLM report generator",
-    "reasoning_summary": "The variant is represented as a missense variation in MC4R. The available source label is uncertain significance. This does not establish clinical causality without transcript-level validation, allele frequency, segregation, phenotype match, functional data, and ACMG/AMP evidence review.",
-    "limitations": "Research record only. Not a clinical diagnosis."
-  },
-  "evidence_examples": {
-    "gene_function": "MC4R functions in the leptin-melanocortin pathway and regulates energy homeostasis.",
-    "source_ids": ["UniProt:P32245", "PMID:12646665", "PMID:32327598", "PMID:33858992"],
-    "structure_context": "Use only when transcript, protein accession, and residue mapping can be reconciled."
-  },
-  "guard_status": {
-    "citation_check": "pass",
-    "abstention_allowed": true,
-    "clinical_use": "not permitted"
-  }
-}
-```
-
-Full guarded report examples and summaries are included in the repository under `reports/full_reports/`.
-
-## Limitations
-
-- The current evidence panel is limited and is not a genome-wide production knowledge base.
-- Missense prediction remains below specialist predictors such as REVEL and AlphaMissense.
-- Broad score model performance is partly driven by easier consequence classes.
-- Structure evidence is only used when transcript/protein/residue mapping can be reconciled.
-- Exact historical versions for some external resources require final manual verification before manuscript submission.
-- The model has not been prospectively validated in clinical settings.
-- Generated text can still contain errors and must be reviewed by domain experts.
-- The uploaded artifact is an adapter, not a standalone full model.
-
-## How to Load the Adapter
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from peft import PeftModel
-import torch
-
-base_model = "meta-llama/Llama-3.1-8B-Instruct"
-adapter_model = "Babajaan/KAU-BioMedLLM"
-
-tokenizer = AutoTokenizer.from_pretrained(base_model)
-model = AutoModelForCausalLM.from_pretrained(
-    base_model,
-    torch_dtype=torch.float16,
-    device_map="auto",
-)
-model = PeftModel.from_pretrained(model, adapter_model)
-model.eval()
-
-messages = [
-    {
-        "role": "user",
-        "content": (
-            "For research use only, generate a structured biomedical variant "
-            "interpretation report for BRCA1 c.5096G>A p.Arg1699Gln. "
-            "Include uncertainty and do not provide clinical advice."
-        ),
-    }
-]
-
-prompt = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True,
-)
-inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
-
-with torch.no_grad():
-    outputs = model.generate(
-        **inputs,
-        max_new_tokens=700,
-        temperature=0.2,
-        do_sample=True,
-    )
-
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
-
-## Compute Infrastructure
-
-The project was developed and evaluated using the King Abdulaziz University Aziz High Performance Computing environment. The workflow used the Aziz HPC software environment, PBS job scheduling, and GPU resources including A100 GPU jobs where available.
-
-KAU Aziz HPC support resources:
-
-- Aziz Supercomputer Support Center: https://hpcc-kau.com/support/
-- PBS and GPU job support documentation are available through the Aziz HPC knowledge base.
-
-## Ethical and Safety Considerations
-
-Biomedical LLM systems can produce fluent but incorrect explanations. KAU-BioMedLLM therefore uses a guard-first design:
-
-- numeric score and LLM narrative are separated;
-- citations are required for biomedical claims;
-- reports can abstain when evidence is insufficient;
-- outputs are research-use only;
-- no clinical decision should be made from model output.
-
-Users should treat all outputs as draft research interpretations requiring expert review.
-
-## Citation and Acknowledgement
-
-If you use this model or the associated code/reports, please cite the forthcoming project manuscript or repository when available.
-
-Manuscript status: in preparation. A formal citation will be added after manuscript submission or publication.
-
-Suggested acknowledgement:
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
 
-> This work used computational resources from the King Abdulaziz University Aziz High Performance Computing environment.
+## Model Card Contact
 
-## License and Base Model Notes
+[More Information Needed]
+### Framework versions
 
-This repository contains a LoRA adapter and project documentation. It does not redistribute the full Llama base model. Use of the adapter with `meta-llama/Llama-3.1-8B-Instruct` is subject to the Meta Llama 3.1 Community License and Acceptable Use Policy. Qwen-derived prototype artifacts are subject to Qwen license terms. Users are responsible for verifying license compatibility for their own use case.
+- PEFT 0.19.1