Procedural Memory Augmented Deep Reinforcement Learning
Ying Ma, Joseph C. Brooks, Hongming Li, José Carlos Príncipe · IEEE Transactions on Artificial Intelligence · 2020
Inspired by the human brain, we propose an external memory-augmented decision-making architecture for video processing. A self-organizing object detector is employed as a frontend to deconstruct the environment. This is done by extracting events from the flow of time and detecting objects within the frames. By employing an extra working memory where objects are temporarily stored, the system can extract properties of the stored objects related to the task. We propose a deep reinforcement learning (RL) neural network to learn affordances, i.e., a sequence of actions to manipulate these objects. The RL network and object detector are trained alternatively. After both the network and detector are trained, the objects and their affordances are transferred to an external memory. They are then utilized when the same objects are detected in input frames. Here, we use a combination of a dictionary and a linked list for the external memory that can be accessed by either content or temporal order. This dual access is motivated by the temporal property of human procedural memory. The proposed memory-augmented RL framework brings advantages of transferability, explainability and computational efficiency with respect to conventional deep learning architectures. We validate the framework on the video game Super Mario Brothers to show superiority to some classical deep RL architectures and exemplify these three advantages.