Crowd-Powered Hybrid Classification Services: Calibration is all you need
Burcu Sayin, Evgeny Krivosheev, Jorge Ramírez, Fabio Casati, Таран Екатерина Александровна, Вероника Анатольевна Маланина, Jie Chi Yang · 2021
Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.