HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale
Sepideh Mesbah, Inès Arous, Jie Chi Yang, Alessandro Bozzon · 2023
Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts’ ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers’ evaluation is, however, less reliable and might substantially differ from experts’ evaluation.