The Promise and Limitation of Explainable AI in Smart Cities: A Sociotechnical Perspective
Naomi Berenfeld, Ning Nan, CARSON C. WOO · Academy of Management Proceedings · 2025
Artificial Intelligence tools (AITs) designed for multiuser systems, such as smart cities, often cause a conflict between the fairness perceived by individual users and the group goal embedded into the AI algorithm. This study investigates how explainable AI (XAI) affects user collaboration with AITs in smart cities when fairness conflicts arise between individual and group goals. Drawing on Fairness Heuristics Theory (FHT), we conducted scenario surveys to assess users' intentions to collaborate with smart city AITs while facing the fairness conflict. Our results indicate that XAI's effects vary depending on the specific context and the nature of XAI provided. In a multiuser system like a smart city, when XAI emphasizes group-level prioritization, it can exacerbate the individual user’s fairness conflicts, negatively affecting user-AIT collaboration. Conversely, it can enhance user-AIT collaboration by highlighting individual user features or supported societal goals. Furthermore, there is currently no empirical evidence regarding the effect of XAI during emergencies. These findings underscore both XAI's potential benefits and limitations in promoting user collaboration within multiuser systems. This research contributes to the expanding XAI literature. It also provides insights that can assist practitioners in designing AI technologies that enhance user collaboration in complex environments, such as smart cities. Acknowledgements: This research is supported by the Social Sciences and Humanities Research Council of Canada (SSHRC) Grant #430- 2022-00504. In addition, this work is supported in part by the Institute for Computing, Information and Cognitive Systems (ICICS) at UBC.