Adaptation for Changing Stochastic Environments through Online POMDP Policy Learning
Guy Shani, Ronen I. Brafman, Solomon Eyal Shimony · 2005
Abstract. Computing optimal or approximate policies for partially observable Markov decision processes (POMDPs) is a difficult task. When in addition the characteristics of the environment change over time, the problem is further com-pounded. A policy that was computed offline may stop being useful after suffi-cient changes to the environment have occurred. We present an online algorithm for incrementally improving POMDP policies, that is highly motivated by the Heuristic Search Value Iteration (HSVI) approach — locally improving the cur-rent value function after every action execution. Our algorithm adapts naturally to slow changes in the environment, without the need to explicitly model the changes. In initial empirical evaluation our algorithm shows a marked improve-ment over other online POMDP algorithms. 1