Multi-agent generative AI for concept evaluation: consistency, knowledge integration and human alignment
Mas’udah Mas’udah, Pavel Livotov, Björn Ragnar Kokoschko, Wanyu Xu, Immanuel Hendra, Niklas Hartmann · Proceedings of the Design Society · 2026
ABSTRACT: Early-stage concept evaluation is critical for selecting viable designs. This study introduces a multi-agent generative AI framework for assessing concepts across four configurations: AI with retrieval-augmented knowledge, AI without external knowledge, human experts, and a hybrid approach. The findings show that AI panels tend to produce uniform evaluation patterns, while retrieval-augmented knowledge alters rating behaviour without leading to closer alignment with human judgement. Hybrid setting achieved closest alignment, indicating AI is effective when combined with expert interpretation.