Robot-learning using a Tree-based Policy Representation
Martijn J.A. Zeestraten · Research Repository (Delft University of Technology) · 2013
Learning is an important aspect in creating versatile robots. Pre-programming a robot to acquire a wide variety of skills in an ever changing environment is unfeasible. Robot learning provides a promising alternative. Two well-established learning techniques are Programming by Demonstration (PbD) and Learning from Exploration (LfE). PbD and LfE are often combined to strengthen each other. PbD is used because it allows fast learning: with only a few demonstrations, robots are able to reproduce tasks with reasonable performance. After these demonstrations, LfE is used to improve the robot's task performance or to adjust this skills to changing environments. Robots often use continuous mappings between states and actions to represent a skill. Such mappings are called policies and are represented by function approximators. The shape of the policy is determined by the parameters. During learning the robot tries to find optimal parameters for the policy. As the complexity of the skill increases, the number of parameters required to accurately describe the policy for this skill also increases. As the number of parameters increases, the complexity of the solution space increases as well. It is most likely that the LfE algorithm requires more trials to converge for these complex search spaces than simpler search spaces, thus the LfE performance decreases as the complexity of the search space increases. In this thesis a novel multi-resolution policy representation is investigated. The method, called Tree-based policy representation, creates a multi-resolution model based on demonstration data. After this initialization, LfE can use the structure of the Multi-resolution policy to increase learning performance. The method is tested in multiple experimental scenarios. The Tree-based policy representation achieves better learning performance compared to conventional `flat' policy representations, when learning motions that clearly have a multi-resolution aspect. In other cases, the Tree-based movement representation performs equally well or worse compared to standard `flat' policy representations.