Mastering Uncertainty with a Sense of Control. Incorporating a Human-Based Sense of Control in Reinforcement Learning Agents for Uncertain Dynamic Environments
Annika Österdiekhoff · Publikationen an der Universität Bielefeld (Universität Bielefeld) · 2026
Reinforcement learning agents excel in game applications, such as Chess and Minecraft, but face challenges in highly dynamic and uncertain environments. For instance, in serial multitasking scenarios involving multiple, concurrent tasks, agents often have difficulty determining when to change strategies or tasks. Additionally, they may struggle to prioritize tasks effectively and have issues with identifying noise in the outcomes of their actions. A cognitive science perspective suggests that a sense of control arises in humans, which helps them navigate such challenges. This sense emerges from sensorimotor processes that involve predicting future events and comparing them to actual outcomes, as well as from the experience of exerting control. A decrease in the sense of control, as indicated through sensorimotor comparisons, indicates uncertainty regarding action outcomes and can lead to adjustments in strategies. Furthermore, anticipating the action control required for a task shapes this sense of control. These insights suggest that incorporating a sense of control into reinforcement learning agents could enhance their performance. Therefore, this thesis introduces the first computational model of a sense of control integrated into reinforcement learning agents, aiming to enhance their performance in navigating dynamic, uncertain environments. Initial research, drawing on human studies, examines the sense of control in dynamic and uncertain single-task and multitasking environments using a game-like action control task. Findings show that unpredictable uncertainty impacts the sense of control in single-task settings. In multitasking, individuals develop a task-specific sense of control for each subtask, forming an overall feeling of control. The study also connects this overall sense of control to task switching behaviors when tasks share similar characteristics. Drawing from insights gained from human studies, this work proposes a mathematical model of the sense of control, comprising two components: (1) an evaluative indicator reflecting the predictability of action outcomes, and (2) a predictive indicator assessing the control needed to address the current situation within a task. The computational model of the sense of control is integrated into reinforcement learning agents and evaluated in single-task and multitask settings. Results show no significant differences between agents with and without a sense of control in single-task environments, but a marked improvement in multitasking scenarios. Agents with a sense of control outperform those without it and even surpass human participants. These agents prioritize more complex tasks, switch more frequently, and make transitions at optimal times, leading to improved performance. The integration of cognitive mechanisms, inspired by human behavior, into reinforcement learning agents demonstrated significant performance improvements, particularly in complex and dynamic environments like multitasking scenarios. This interdisciplinary approach, combining cognitive science with artificial intelligence, enhances the applicability of artificial intelligence technologies across various domains and contributes to performance advancements.