Predictive Representations of State
Michael L. Littman, Richard S. Sutton · 2001
We show that states of a dynamical system can be usefully repre-sented by multi-step, action-conditional predictions of future ob-servations. State representations that are grounded in data in this way may be easier to learn, generalize better, and be less depen-dent on accurate prior models than, for example, POMDP state representations. Building on prior work by Jaeger and by Rivest and Schapire, in this paper we compare and contrast a linear spe-cialization of the predictive approach with the state representa-tions used in POMDPs and in k-order Markov models. Ours is the rst specic formulation of the predictive idea that includes both stochasticity and actions (controls). We show that any system has a linear predictive state representation with number of predictions no greater than the number of states in its minimal POMDPmodel. In predicting or controlling a sequence of observations, the concepts of state and