Add live demo link

33f2f88 verified 6 days ago

5.64 kB

base_model: Qwen/Qwen3.5-0.8B
license: apache-2.0
language:
  - en
library_name: peft
pipeline_tag: text-generation
datasets:
  - hetanshwaghela/autoscientist-healthcare-reasoning
tags:
  - healthcare
  - medical
  - clinical-reasoning
  - chain-of-thought
  - grounded-generation
  - hallucination-mitigation
  - safety
  - adaptive-data
  - autoscientist
  - lora
  - qwen
  - fine-tuned

🧪 AutoScientist — Healthcare Clinical-Reasoning (LoRA) + Experiment Log

Built with Adaptive Data by Adaption. A Qwen/Qwen3.5-0.8B LoRA adapter trained on the adapted clinical-reasoning dataset — released together with the honest, reproducible experiment that produced it.

TL;DR — read this first (integrity over hype)

This is a research artifact, not a leaderboard-beating model. On a held-out LLM-judged win-rate (Gemini 3.1 pro, 200 samples) the base model wins, 58/42. We report that plainly. The contribution of this project is the dataset (+30% quality, Grade B→A — see the dataset card) and the rigorous finding documented below.

📓 The AutoScientist's notebook

Hypothesis. Adapting a high-quality medical chain-of-thought dataset (+30% platform quality) and SFT-ing a small model on it should beat the base model on held-out medical reasoning.

Experiment 1 — 20k rows, LoRA r=64, 3 epochs. → Base 58 / adapted 42. Medical category: base 55 / adapted 46. → Diagnostic: eval-loss plateaued after ~epoch 1 (1.559→1.525) while train-loss kept falling (2.27→1.19) — overfitting.

Experiment 2 — controlled follow-up: 60k rows, +40% general-purpose data (to counter catastrophic forgetting, per platform guidance), tuned recipe, more steps. → Base 62 / adapted 38. It got worse — higher peak LR + more steps moved the model further from a strong base.

Conclusion (reproducible across two runs).

Supervised fine-tuning an already-instruction-tuned 0.8B model on long (~780-word) chain-of-thought makes it more verbose, and the judge prefers the base's crisper answers. Dataset quality and small-model win-rate are different axes. No hyperparameter or data-mix change flipped it — the effect is structural, not a bug.

This is the result the challenge is designed to surface: a clean, honest, reproducible negative — the dataset is the win; the model is the documented experiment.

Model description

Base model: Qwen/Qwen3.5-0.8B — the exact model AutoScientist trained from (served via togethercomputer/Qwen3.5-0.8B). All numbers are relative to this base, identical prompts and settings.
Method: AutoScientist, LoRA SFT, train_on_inputs=false.
Released recipe (Experiment 1, the stronger of the two): r=64, α=128, dropout 0.05, target q,k,v,o_proj, 3 epochs, LR ~1.1e-4 (cosine, warmup 0.05), weight-decay 0.01, grad-clip 2. Final train-loss ≈ 1.19, eval-loss ≈ 1.53 (168 steps).
Weight format: LoRA adapter (adapter_model.safetensors + adapter_config.json), base repointed to Qwen/Qwen3.5-0.8B for portable PEFT loading.
Language: English. Size: ~0.8B base params.

📊 Evaluation (base vs. adapted) — honest

Judge = Gemini 3.1 pro, 200 held-out samples, identical prompts/settings.

Win-rate (head-to-head)	Base `Qwen3.5-0.8B`	Adapted (this model)
On the dataset task	58	42
Medical category (all tasks)	55	46

Dataset quality (Adaptive Data, platform-measured): +30% overall, Grade B→A, completion quality +37.9%, message quality +17.6%, percentile 15.3→33.0.

🩺 What it's designed to do (safety blueprint)

Trained to reason then answer, stay grounded in evidence, hedge, recommend clinician confirmation, escalate red-flag/emergency symptoms, refuse when uncertain, and preserve numeric values exactly. (Design goals of the dataset; not a claim of clinical accuracy.)

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen3.5-0.8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base)
model = PeftModel.from_pretrained(model, "hetanshwaghela/autoscientist-healthcare-reasoning")

⚠️ Limitations & safety

Underperforms its base on the held-out judge — do not treat as an improvement over Qwen/Qwen3.5-0.8B. Tends to be verbose.
Educational decision-support, not a medical device. Not for individual diagnosis, treatment, or dosing without a qualified clinician. Escalate emergencies; preserve numeric values exactly.
Inherits base-model and machine-generated-data limitations; English, exam-style skew.

🔁 Reproducibility

Base: Qwen/Qwen3.5-0.8B (served as togethercomputer/Qwen3.5-0.8B)
AutoScientist experiment id: 0eac225d-a25c-43b3-ae85-0549d5d08d8e
Adapted dataset id: 26048b57-f164-46d5-810b-12d498a76660 (20,000 rows)
Dataset: https://huggingface.co/datasets/hetanshwaghela/autoscientist-healthcare-reasoning
Kaggle model: https://www.kaggle.com/models/hetanshwaghela1/autoscientist-healthcare-reasoning
🧪 Live demo (HF Space): https://huggingface.co/spaces/hetanshwaghela/autoscientist-healthcare-demo

Credit

Built with Adaptive Data by Adaption. Base: Qwen/Qwen3.5-0.8B. Foundation data: FreedomIntelligence/medical-o1-reasoning-SFT (Apache-2.0). Public-health blend: CDC public-domain text (Source: Centers for Disease Control and Prevention; no CDC endorsement is implied).