Design Patterns for Machine Learning-Based Systems With Humans in the Loop
Jakob Smedegaard Andersen, Walid Maalej · IEEE Software · 2023
The development and deployment of systems using supervised machine learning (ML) remain challenging: mainly due to the limited reliability of prediction models and the lack of knowledge on how to effectively integrate human intelligence into automated decision-making. Humans involvement in the ML process is a promising and powerful paradigm to overcome the limitations of pure automated predictions and improve the applicability of ML in practice. We compile a catalog of design patterns to guide developers select and implement suitable humanin-the-loop (HiL) solutions. Our catalog takes into consideration key requirements as the cost of human involvement and model retraining. It includes four training patterns, four operation patterns, and two orthogonal cooperation patterns.