Near-optimal intelligent control for continuous set-point regulator problems via approximate dynamic programming
Augustine O. Esogbue, Warren E. Hearnes · 1999
Optimization theory provides a framework for determining the best decisions or actions with respect to some mathematical model of a process. This research focuses on learning to act in a near-optimal manner for problems that either have no model or the model is too complex to solve by traditional methods. Learning to act or to make a near-optimal decision based on past experience may be formulated as a reinforcement learning problem. One approach to solving this class of problems is via approximate dynamic programming methods. The convergence properties of these methods, however, are established primarily for the case of discrete state and action spaces. A primary objective of this research, therefore, is to develop efficient methods of learning which act in complex systems with continuous state and action spaces. In the absence of transition probabilities, Monte-Carlo approaches are employed to estimate function values in an iterative, incremental procedure. Derivative-free line search methods are used to obtain a near-optimal action in the continuous action space for a discrete subset of the state space. This near-optimal control policy is then extended to the entire continuous state space via a fuzzy additive model. To compensate for approximation errors, a modified procedure for perturbing the generated control policy is developed. Convergence results under moderate assumptions and stopping criteria are established. To aid in the successful control of more complex systems, a hierarchical controller is proposed that decomposes the control problem into interdependent subproblems. Practical experiments on a class of set-point regulator problems illustrate the properties and benefits of the proposed algorithms.