Generalizability of Coronavirus Disease 2019 (COVID-19) Clinical Prediction Models
Shubhada Hooli, Carina J. King · Clinical Infectious Diseases · 2020
To the Editor—The recent article by Sun et al compares exposure, demographic, clinical, and diagnostic test characteristics between coronavirus disease 2019 (COVID-19) polymerase chain reaction (PCR)–confirmed cases and PCR-negative cases evaluated at the designated screening and referral hospital in Singapore [1]. The authors then present 4 COVID-19 case prediction models. We question the reproducibility of their results. Multivariable logistic regression models can be overfitted to their derivation sample when the predictor to outcome of interest ratio is > 1:10. Overfitting a logistic regression model can lead to spuriously high area under the receiver operating curve implying good model discrimination. However, this limits the generalizability applied to another population. Each of the models violate this principle (Table 1), although we note that model 4, with the poorest performance, was close to meeting this criterion. Characteristics of Coronavirus Disease 2019 Case Prediction Models Abbreviations: AUC, area under the curve. Characteristics of Coronavirus Disease 2019 Case Prediction Models Abbreviations: AUC, area under the curve. Nearly every nation has limited testing resources in the face of this rapidly progressing pandemic. Case identification tools could play a crucial role in containment and mitigation strategies. This is why it is extremely important that models be designed with a focus on generalizability of findings, which includes clearly defined predictors. When describing the clinical characteristics of patients included in the models, the authors do not provide sufficient detail for others to replicate and externally validate their tool, with descriptors such as “elevated body temperature” and “elevated respiratory rate.” No other disease process, in recent memory, has captured the world’s attention like COVID-19. Appropriately, scientists are racing to better understand and mitigate this global pandemic; a preprint review uploaded on 27 March 2020 already identified 19 COVID-19 prediction models and also raised concerns about the quality of these tools [2]. During such dire circumstances, more than ever we must be vigilant to uphold our own standards. Potential conflicts of interest. The authors: No reported conflicts of interest. Both authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.