Learning Visual Routines with Reinforcement Learning
Andrew Kachites McCallum · 1996
Reinforcement learning is an ideal framework to learn visual routines since the routines are made up of sequences of actions. However, such algorithms must be able to handle the hidden state (perceptual aliasing) that results from visual routine's purposefully narrowed attention. The U-Tree algorithm successfully learns visual routines for a complex driving task in which the agent makes eye movements and executes deictic actions in order to weave in and out of traffic on a four-laned highway. The task involves hidden state, time pressure, stochasticity, a large world state space, and a large perceptual state space. U-Tree uses a tree-structured representation, and is related to work on Prediction Suffix Trees (Ron, Singer, & Tishby 1994), Parti-game (Moore 1993), Galgorithm (Chapman & Kaelbling 1991), and Variable Resolution Dynamic Programming (Moore 1991). UTree is a direct descendant of Utile Suffix Memory (McCallum 1995c), which used short-term memory, but not selective perception...