An analysis of feature-based and state-based representations for module-based learning in mobile robots
Esther Luna Colombini, Carlos H. C. Ribeiro · 2005
The information available to robots in real tasks is widely distributed both in time and space, requiring the agent to search for relevant information. In this paper, we implement a solution that uses qualitative and quantitative knowledge to turn robot tasks able to be treated by reinforcement learning (RL) algorithms. The steps of this procedure include: 1) to decompose the overall task into smaller ones, using abstraction and macro-operators, thus achieving a discrete action space; 2) to use observation functions of the environment - here called features - to achieve both time and state space discretisation; 3) to use quantitative knowledge to design controllers that are able to solve the subtasks; 4) to learn the coordination of these behaviours using RL, more specifically Q-learning. The approach was verified on an increasingly complex set of robot tasks using a Khepera robot simulator. Two approaches for space discretisation were used, one based on features and the other on states. The learned policies over these two models were compared to a predefined hand-crafted one. It was found that the learned policy over the state-based discretisation leads quickly to good results, although it can not be applied to complex tasks, where the state space representation becomes computationally unfeasible.