Toward expert-level medical text validation with language models
Asad Aali, Vasiliki Bikia, Maya Varma, Nicole Chiou, Sophie Ostmeier, Arnav Singhvi, Magdalini Paschali, Ashwin Kumar, Andrew Johnston, Karimar Amador-Martinez, Eduardo Guerrero, Paola Naovi Cruz Rivera, Sergios Gatidis, Christian Blüthgen, Eduardo Pontes Reis, Eddy D. Zandee van Rilland, Poonam Hosamani, Kevin Keet, Minjoung Go, Evelyn Ling · npj Digital Medicine · 2026
With the growing use of language models (LMs) in clinical environments, there is an immediate need to evaluate the accuracy of LMs. Detecting errors in LM-generated text is challenging because (1) manual review is costly and (2) expert-composed reference outputs are often unavailable in real-world settings. While the “LLM-as-a-judge” paradigm (a LM evaluating another LM) offers scalable evaluation, even frontier LMs can miss subtle but clinically significant errors. To address these challenges, we propose MedVAL, a novel, self-supervised, data-efficient distillation method that leverages synthetic data to train evaluator LMs to assess whether LM-generated medical outputs are factually consistent with inputs, without requiring physician labels or reference outputs. To evaluate LM performance, we introduce MedVAL-Bench, a dataset of 840 physician-annotated outputs across 6 diverse clinical use cases capturing real-world challenges. Each output is reviewed following a physician-defined taxonomy of risk levels and error categories, enabling evaluation of LMs in making deployment safety decisions. Across 10 state-of-the-art LMs spanning open-source, proprietary, and medically adapted models, MedVAL distillation significantly improves ( p < 0.001) alignment with physicians across seen and unseen tasks, increasing average F1 scores from 66% to 83%. Despite strong baseline performance, MedVAL improves the best-performing proprietary LM (GPT-4o) by 8% without training on physician-labeled data, demonstrating a performance statistically non-inferior to a single human expert on a subset annotated by multiple physicians ( p < 0.001). To support a scalable pathway towards clinical integration, we open-source: (1) Codebase , (2) MedVAL-Bench , (3) MedVAL-4B . Across evaluated tasks, our study provides evidence of LMs approaching expert-level ability in risk-aware validation of LM-generated medical text.