Experiment-log model card (honest)

README.md

@@ -3,7 +3,7 @@ base_model: Qwen/Qwen3.5-0.8B
 license: apache-2.0
 language:
 - en
-library_name: peft            # LoRA adapter (apply on the base model)
+library_name: peft
 pipeline_tag: text-generation
 datasets:
 - hetanshwaghela/autoscientist-healthcare-reasoning
@@ -22,98 +22,99 @@ tags:
 - fine-tuned
 ---
 
-# AutoScientist — Healthcare Clinical-Reasoning Model (LoRA)
+# 🧪 AutoScientist — Healthcare Clinical-Reasoning (LoRA) + Experiment Log
 
 > **Built with Adaptive Data by Adaption.**
+> A `Qwen/Qwen3.5-0.8B` LoRA adapter trained on the adapted clinical-reasoning dataset —
+> released together with the *honest, reproducible experiment* that produced it.
 
-A **Qwen3.5-0.8B** LoRA adapter fine-tuned with **AutoScientist** on an adapted
-healthcare **clinical-reasoning** dataset (reason-then-answer chain-of-thought), plus a
-grounded public-health operations blend.
+## TL;DR — read this first (integrity over hype)
 
-## ⚠️ Honest performance summary (read first)
+This is a **research artifact**, not a leaderboard-beating model. On a held-out
+LLM-judged win-rate (Gemini 3.1 pro, 200 samples) the **base model wins, 58/42**.
+We report that plainly. The **contribution of this project is the dataset** (+30%
+quality, Grade B→A — see the
+[dataset card](https://huggingface.co/datasets/hetanshwaghela/autoscientist-healthcare-reasoning))
+and the **rigorous finding** documented below.
 
-This is a **research artifact**, released for reproducibility and to accompany the
-adapted dataset — **not** a model that outperforms its base.
+## 📓 The AutoScientist's notebook
 
-- On a held-out LLM-judged win-rate (Gemini 3.1 pro, 200 held-out samples), the
-  **base model wins**: **base 58 / adapted 42** on the dataset task, **base 55 /
-  adapted 46** on the broader Medical category.
-- The base `Qwen/Qwen3.5-0.8B` is already instruction-tuned; SFT-ing a 0.8B model on
-  long chain-of-thought completions makes it more verbose, and the judge prefers the
-  base's more concise answers. Two independent training runs both landed below base.
-- **The strong result in this project is the *dataset*, not the model** — see the
-  [dataset card](https://huggingface.co/datasets/hetanshwaghela/autoscientist-healthcare-reasoning)
-  for the +30% Adaptive Data quality improvement (Grade B→A). This model is the
-  companion training artifact.
+**Hypothesis.** Adapting a high-quality medical chain-of-thought dataset (+30% platform
+quality) and SFT-ing a small model on it should beat the base model on held-out medical
+reasoning.
+
+**Experiment 1** — 20k rows, LoRA r=64, 3 epochs.
+→ Base **58** / adapted **42**. Medical category: base **55** / adapted **46**.
+→ Diagnostic: eval-loss plateaued after ~epoch 1 (1.559→1.525) while train-loss kept
+falling (2.27→1.19) — **overfitting**.
+
+**Experiment 2** — controlled follow-up: **60k rows**, **+40% general-purpose data**
+(to counter catastrophic forgetting, per platform guidance), tuned recipe, more steps.
+→ Base **62** / adapted **38**. It got **worse** — higher peak LR + more steps moved the
+model *further* from a strong base.
+
+**Conclusion (reproducible across two runs).**
+> Supervised fine-tuning an *already-instruction-tuned* 0.8B model on long (~780-word)
+> chain-of-thought makes it **more verbose**, and the judge prefers the base's crisper
+> answers. **Dataset quality and small-model win-rate are different axes.** No
+> hyperparameter or data-mix change flipped it — the effect is structural, not a bug.
+
+This is the result the challenge is designed to surface: a clean, honest, reproducible
+negative — the dataset is the win; the model is the documented experiment.
 
 ## Model description
 - **Base model:** `Qwen/Qwen3.5-0.8B` — the exact model AutoScientist trained from
-  (served via `togethercomputer/Qwen3.5-0.8B`). All numbers above are **relative to
-  this base** with identical prompts and inference settings.
-- **Fine-tuning method:** AutoScientist, LoRA (SFT, `train_on_inputs=false`).
-- **Recipe (as run):** rank `r=64`, `alpha=128`, `dropout=0.05`, target modules
-  `q_proj,k_proj,v_proj,o_proj`, **3 epochs**, learning rate ~`1.1e-4` (cosine,
-  warmup 0.05), weight decay 0.01, grad-clip 2. Final train loss ≈ 1.19,
-  eval loss ≈ 1.53 (168 steps).
-- **Weight format:** **LoRA adapter** (`adapter_model.safetensors` +
-  `adapter_config.json`). Apply on top of the base model with PEFT.
-- **Language:** English. **Size:** ~0.8B base params (small — fast, publishable).
-
-## Training data
-Trained on the **Adapted Healthcare Clinical-Reasoning (AutoScientist)** dataset — the
-adapted output of the foundation → Adaptive Data → AutoScientist pipeline. Foundation:
-[`FreedomIntelligence/medical-o1-reasoning-SFT`](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT)
-(Apache-2.0, gold chain-of-thought) blended with 30 curated CDC-grounded public-health
-rows; adapted with `deduplication` + `hallucination_mitigation` under a
-clinical-reasoning safety blueprint. See the dataset card for full provenance.
-
-- HF dataset: https://huggingface.co/datasets/hetanshwaghela/autoscientist-healthcare-reasoning
-- Kaggle dataset: https://www.kaggle.com/datasets/hetanshwaghela1/autoscientist-healthcare-reasoning
-
-Recorded for reproducibility / baseline comparison:
-- Base model id: `Qwen/Qwen3.5-0.8B` (served as `togethercomputer/Qwen3.5-0.8B`)
-- AutoScientist experiment id: `0eac225d-a25c-43b3-ae85-0549d5d08d8e`
-- Adapted dataset id: `26048b57-f164-46d5-810b-12d498a76660` (20,000 rows)
-
-## Evaluation results (base vs. adapted)
-Held-out LLM-judged win-rate, judge = **Gemini 3.1 pro**, 200 held-out samples,
-identical prompts and settings for both models.
+  (served via `togethercomputer/Qwen3.5-0.8B`). All numbers are relative to this base,
+  identical prompts and settings.
+- **Method:** AutoScientist, LoRA SFT, `train_on_inputs=false`.
+- **Released recipe (Experiment 1, the stronger of the two):** r=64, α=128, dropout 0.05,
+  target `q,k,v,o_proj`, 3 epochs, LR ~1.1e-4 (cosine, warmup 0.05), weight-decay 0.01,
+  grad-clip 2. Final train-loss ≈ 1.19, eval-loss ≈ 1.53 (168 steps).
+- **Weight format:** **LoRA adapter** (`adapter_model.safetensors` + `adapter_config.json`),
+  base repointed to `Qwen/Qwen3.5-0.8B` for portable PEFT loading.
+- **Language:** English. **Size:** ~0.8B base params.
+
+## 📊 Evaluation (base vs. adapted) — honest
+Judge = **Gemini 3.1 pro**, 200 held-out samples, identical prompts/settings.
 
 | Win-rate (head-to-head) | Base `Qwen3.5-0.8B` | Adapted (this model) |
 |---|---|---|
 | On the dataset task | **58** | 42 |
 | Medical category (all tasks) | **55** | 46 |
 
-**Interpretation:** the adapted model does **not** beat base on this held-out judge.
-The value of the pipeline shows up in **dataset quality** (see dataset card: +30%,
-Grade B→A, top-third percentile), not in the small-model head-to-head.
+**Dataset quality (Adaptive Data, platform-measured):** +30% overall, Grade B→A,
+completion quality +37.9%, message quality +17.6%, percentile 15.3→33.0.
 
-## Intended uses
-- Reproducing the AutoScientist pipeline; research on grounding, hallucination
-  mitigation, CoT reasoning, and small-model fine-tuning behavior.
-- Clinical-reasoning **education**, with the caveats below.
-
-## Limitations & safety
-- **Underperforms its base model on the held-out judge** (see above). Do not treat it
-  as an improvement over `Qwen/Qwen3.5-0.8B`.
-- **Educational decision-support, not a medical device.** Not for individual diagnosis,
-  treatment, or medication dosing without a qualified clinician.
-- Inherits limitations/biases of the base model and of partly machine-generated data;
-  tends to be verbose. English, exam-style skew.
-- Escalate emergencies; do not rely on outputs for high-stakes decisions; preserve
-  numeric values exactly.
+## 🩺 What it's designed to do (safety blueprint)
+Trained to *reason then answer*, stay grounded in evidence, hedge, recommend clinician
+confirmation, escalate red-flag/emergency symptoms, refuse when uncertain, and preserve
+numeric values exactly. (Design goals of the dataset; not a claim of clinical accuracy.)
 
 ## How to use
 ```python
 from peft import PeftModel
 from transformers import AutoModelForCausalLM, AutoTokenizer
-
 base = "Qwen/Qwen3.5-0.8B"
 tok = AutoTokenizer.from_pretrained(base)
 model = AutoModelForCausalLM.from_pretrained(base)
 model = PeftModel.from_pretrained(model, "hetanshwaghela/autoscientist-healthcare-reasoning")
 ```
 
+## ⚠️ Limitations & safety
+- **Underperforms its base on the held-out judge** — do not treat as an improvement over
+  `Qwen/Qwen3.5-0.8B`. Tends to be verbose.
+- **Educational decision-support, not a medical device.** Not for individual diagnosis,
+  treatment, or dosing without a qualified clinician. Escalate emergencies; preserve
+  numeric values exactly.
+- Inherits base-model and machine-generated-data limitations; English, exam-style skew.
+
+## 🔁 Reproducibility
+- Base: `Qwen/Qwen3.5-0.8B` (served as `togethercomputer/Qwen3.5-0.8B`)
+- AutoScientist experiment id: `0eac225d-a25c-43b3-ae85-0549d5d08d8e`
+- Adapted dataset id: `26048b57-f164-46d5-810b-12d498a76660` (20,000 rows)
+- Dataset: https://huggingface.co/datasets/hetanshwaghela/autoscientist-healthcare-reasoning
+- Kaggle model: https://www.kaggle.com/models/hetanshwaghela1/autoscientist-healthcare-reasoning
+
 ## Credit
 **Built with Adaptive Data by Adaption.** Base: `Qwen/Qwen3.5-0.8B`. Foundation data:
 `FreedomIntelligence/medical-o1-reasoning-SFT` (Apache-2.0). Public-health blend: CDC