ZiqiHao/m31-ehr-future-conditions

Patient-Timeline Transformer — Future Condition Forecasting (M31)

Decoder-only, nanoGPT-style transformer (~3.8M params; 4 layers / 4 heads / d_model 256, context 1024) that predicts which of 40 target conditions are newly diagnosed in the 5 years after a patient's anchor, from their leak-free pre-anchor Synthea EHR timeline. Two stages: self-supervised next-event pretraining then multi-label fine-tuning on the [ANCHOR] hidden state (BCE and focal-loss variants).

• Code & full leak-free pipeline: https://github.com/Ziqi-Hao/m31-ehr-future-conditions
• Technical report: ziqi_hao_results.pdf in that repo.

Results (held-out validation, 365 patients)

Model	macro-AUROC	mAP
Transformer (BCE)	0.759	0.229
Transformer (focal, gamma=2)	0.764	0.230
Submitted 3-way ensemble (0.65 focal + 0.20 BCE + 0.15 LogReg)	0.766	0.240

Honest 5-fold cross-validation of the submitted ensemble: macro-AUROC 0.768 +/- 0.004, mAP 0.205 +/- 0.014 (prevalence-matched random-baseline mAP ~ 0.047).

Files

• pretrained.pt — next-event pretrained backbone
• finetuned.pt — BCE fine-tune (ensemble member)
• finetuned_focal.pt — focal-loss fine-tune (primary member, ensemble weight 0.65)
• vocab.json — tokenizer vocabulary + gap/age bucket sizes (needed to rebuild the model)

Reproduce

Clone the GitHub repo, drop the checkpoints in models/ and vocab.json in artifacts/, then run python src/evaluate.py. Trained only on the provided training split; no external data or pretrained weights.