Understanding efficient reinforcement learning in humans and machines
Sam Blakeman · BIROn (Birkbeck, University of London) · 2021
One of the primary mechanisms thought to underlie action selection in the brain is Reinforcement Learning (RL). Recently, the use of Deep Neural Networks in models of RL (Deep RL) has led to human-level performance on complex reward-driven perceptual-motor tasks. However, Deep RL is persistently criticised for being data inefficient compared to human learning because it lacks the ability to: (1) rapidly learn from new information and (2) transfer knowledge from past experiences. The purpose of this thesis is to form an analogy between the brain and Deep RL to understand how the brain performs these two processes. To investigate the internal computations supporting rapid learning and transfer we use Complementary Learning Systems (CLS) theory. This allows us to focus on the computational properties of key learning systems in the brain and their interactions. We review recent advances in Deep RL and how they relate to the CLS framework. This results in the presentation of two novel Deep RL algorithms, which highlight key properties of the brain that support rapid learning and transfer: the fast learning of pattern-separated representations in the hippocampus, and the selective attention mechanisms of the pre-frontal cortex. External factors in the environment can also impact upon rapid learning and transfer in the brain. We therefore conduct behavioural experiments that investigate how the degree of perceptual similarity between consecutive experiences affects people’s ability to perform transfer. To do this we use naturalistic 2D video games that vary in perceptual features but rely on the same underlying rules. We discuss the results of these experiments with respect to Deep RL, analogical reasoning and category learning. We hope that the analogy formed over the course of this thesis between the brain and Deep RL can inform future research into efficient RL in humans and machines.