Experience-Based Control and Coordination of Autonomous Mobile Systems in Dynamic Environments
Sebastian Buck · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · 2003
Many real-time machine control skills are too complex and laborious to be coded by hand. Preferably, such skills are acquired by learning algorithms. Suitable algorithms should learn automatically and based on experience from interaction with the machine's environment. But unfortunately, typical learning methods for real world machine control tasks have a number of problems: Huge high-dimensional state spaces complicate inductive learning, and it might be difficult to get a sufficient amount of appropriate training data for learning either because it takes too long or because it is extremely difficult to obtain good examples for learning from exploration. Furthermore, most current learning algorithms rely on a discrete MDP-model of the continuous state space, suffer from the incremental summation of errors during learning, and neglect the existence of undesirable states. The idea behind our approach of experience-based control is to exploit trajectories of successful explorations to approximate a value-function for the state space. To overcome the lack of training data we employ a realistic neural simulation of the machine's dynamics and introduce adequate exploration techniques, such as backward exploration, to acquire learning data. The combination of different exploration techniques allows for the integration of various types of initial knowledge and undesirable states can be integrated in the learning model. Since the majority of machine control tasks in technical applications shows deterministic behavior - or at least a unimodal probability distribution with a small variance - it is possible to use a simple projection-function instead of a complex MDP-model that was originally designed for discrete states. Our algorithms operate directly in a continuous state space and perform a number of explorations before we exploit the data. This is the main reason why our approach is robust against the incremental summation of noise which is often encountered in conventional learning algorithms. For the practical and efficient approximation of continuous functions we employ neural networks and networks of radial basis functions. Our methods have successfully been applied to numerous navigation tasks and tasks of situation dependent algorithm-selection.