Integration of sensory modalities for advice in human-robot scenarios
Francisco Cruz · 2018
Robots in domestic environments are receiving more attention, especially in scenarios where they should interact with parent-like trainers for dynamically acquiring and refining knowledge. In learning approaches, a promising extension has been to incorporate an external parent-like trainer into the learning cycle in order to scaffold and speed up the apprenticeship using advice about what actions should be performed for achieving a goal. Different uni modal control interfaces have been proposed that are often quite limited and do not take into account multiple sensor modalities. In this paper, we propose the integration of audiovisual patterns to provide advice to the agent using multi-modal information. In our approach, advice can be given using either speech, gestures, or a combination of both. We introduce a mathematical model to integrate multi modal information from uni-modal modules based on their confidence. Results show that multimodal integration leads to either strengthen or diminish the integrated confidence value in comparison to the uni-modal approaches.