Predicting Unethical Physician Behavior At Scale: A Distributed Computing Framework
Anastasia Keck, M. Pilar Romero, Robert Sandor, Diane Myung-kyung Woodbridge, Paul Intrevado · 2019
As the amount of publicly shared data increases, developing a robust pipeline to stream, store and process data is critical, as the casual user often lacks the technology, hardware and/or skills needed to work with such voluminous data. In this research, the authors employ Amazon EC2 and EMR and Apache Spark to explore 28.5 gigabytes of CMS Open Payments data in an attempt to identify physicians who may have a high propensity to act unethically, owing to significant transfers of wealth from medical companies. A Random Forest Classifier is employed to predict the top decile of physicians who have the highest risk of unethical behavior in the following year, resulting in an F-Score of 91%. The authors also employ an anomaly detection algorithm that correctly identified a high-profile case of a physician leaving his prestigious position, failing to disclose anomalously-large transfers of wealth from medical companies.