The Expert of Oz
Andreas Martin Butz, Annika Kaltenhauser, Malin Eiband · 2020
We present the paradigm of Expert of Oz studies for the formative evaluation of hypothetical AI systems. These studies follow the principle of Wizard of Oz studies but use a human expert for simulating the AI. This allows the experimenter to not only investigate the user's behavior with and reaction to such a system but also to observe and analyze the performance of the expert in the situation at hand. As a consequence, we can learn both about interaction with the hypothetical AI system and about the required AI functionality at the same time and in realistic interaction scenarios.