Benchmarking of pre-training strategies for electronic health record foundation models
Samson J. Mataraso, Shreya D’Souza, David Seong, Eloïse Berson, Camilo A. Espinosa, Nima Aghaeepour · JAMIA Open · 2025
Objective: Our objective is to compare different pre-training strategies for electronic health record (EHR) foundation models. Materials and Methods: We evaluated three approaches using a transformer-based architecture: baseline (no pre-training), self-supervised pre-training with masked language modeling, and supervised pre-training. The models were assessed on their ability to predict both major adverse cardiac events and mortality occurring within 12 months. The pre-training cohort was 405 679 patients prescribed antihypertensives and the fine tuning cohort was 5525 patients who received doxorubicin. Results: Task-specific supervised pre-training achieved superior performance (AUROC 0.70, AUPRC 0.23), outperforming both self-supervised pre-training and the baseline. However, when the model was evaluated on the task of 12-month mortality prediction, the self-supervised model performed best. Discussion: While supervised pre-training excels when aligned with downstream tasks, self-supervised approaches offer more generalized utility. Conclusion: Pre-training strategy selection should consider intended applications, data availability, and transferability requirements.