Spatiotemporal coherent semantic maps with mobile robots
Ιωάννης Κωσταβέλης · 2014
Along the last decades persistent research endeavors in the areas of robotics andartificial intelligence revealed the great challenge of robot navigation, an areathat combines robot mobility with perception of the environment. Moreover,in modern human societies it is of great importance to build up machines thatcan be operated by non specialists or even by technologically illiterate people,such as youngsters or elderly. Therefore, the mobile robots to be released intothe market in the near future should possess, among others, the potential ofproducing meaningful internal perceptual representations of their own environment,capacitating them to cope a range of real-life situations and tasks. Tomake matters even more challenging, when it comes for mapping and navigation,robots should comprehend human concepts about places and objects,to skillfully deploy in human frequented environments. In response to thischallenge, intense research efforts, to build cognitive robots apt to competentlyperceive and understand their surroundings and to cooperate with humans,take place. With this goal in view, semantic mapping with mobile robots can constituteto a holistic solution in response to the aforementioned challenges. Thesemantic mapping is an augmented representation of the robot’s environmentthat -supplementary to the geometrical knowledge- encapsulates characteristicscompatible with human understanding. It provides several algorithmic opportunitiesfor innovative development of applications that will eventually lead tothe human robot interaction. The main objective of the PhD dissertation in handis the construction of accurate and consistent semantic maps facilitating amplerobot deployment in domestic environments.The motivation behind this PhD dissertation has been the observation thatthe plethora of the existing mapping and navigation algorithms are not ableto provide a sufficient representation of the environment in terms of humans’concepts. This is due to the fact that the mapping methods developed sofar focus on the construction of geometrical maps. Although some of thesesolutions proved to be capable of driving robots into specific target positions,they lack of high level cognition attributes, which would allow them to bring the human-robot interaction one step beyond. Aiming to remove this barrier, thisdissertation is oriented towards the direction of developing semantic mappingalgorithms for high level robot navigation. Therefore, this doctoral researchidentified the basic components of the semantic mapping and developed aninnovative solution for each one of them. Due to the fact that the semanticmapping requires an integrated system that comprises several subordinatemodules, a wide range of a algorithms that serve different tasks had to bedesigned and implemented. Within the context of this thesis several algorithmicmodules have been developed including a competent localization algorithm,novel simultaneously localization and mapping strategies, breakthrough place andobject recognition tactics, as well as the integration of all these methods undera time supervised framework able to produce consistent semantic maps. Dueto the fact that each subordinate module comprises an innovative solutionto the respective field, the resulting semantic mapping system constitutesa state of the art solution in the area of conceptual mapping with mobilerobots. The introduced algorithms exploit solely visual and depth sensors,while by combining basic tools from three different scientific areas such asrobotics, computer vision and machine learning the final objective is accomplished.Overall, the main contribution of this thesis to the advancement the stateof the art is the introduction of a stacked map hierarchy of four differenttype of maps, namely a metric, a topological, a labeled sparse topological andan augmented navigation one. Each of these representations accomplishes aunique purpose: (i) the metric is the physical (lower) layer; (ii) the topologicalone contains abstract geometrical information of the environment, i.e. pointclouds registered in a graph of nodes; (iii) the labeled sparse topological oneestablishes the spatiotemporal coherence by associating the respective nodes inthe topometric maps via place labels and geometrical transformations, enablingbidirectional exchange of information among the conceptual and metric mapsand, last, (iv) the augmented navigation map inheres the significance of thedetected places as well as their connectivity relationships, expressed in termsof their transition probability.The thesis in hand has been developed in a hierarchical fashion and can bedivided into four main chapters. The first one comprises a literature survey ofthe existing semantic mapping methods in which an explicit analysis of the sofar developed methods is sought. The insights of the semantic mapping arereviewed, the distinct components encompassing, to give a categorization of therelated literature, are studied the possible applications in mobile robotics areexamined and, lastly, the methods and databases available for benchmarkingare referred. Furthermore, a quality-based taxonomy of the existing semanticmapping methods highlights the dominant attributes such methods retain.More precisely, according to the scale, to which each method is expanded, the metric map could be either a single scene or a progressively created map.Another important attribute a typical semantic mapping method possessesis the existence of the respective topological map, that is an abstraction ofthe explored environment in terms of a graph. The nodes of such a graph areorganized in a geometrical manner, so as to simultaneously preserve conceptualknowledge about the explored places. Moreover, the modalities (single ormultiple visual cues) utilized to reason about the observed scene constitute anelement apt to distinguish the abundance of different methods. An additionalfeature in many recent semantic mapping techniques is the temporal coherencesuch a map reveals, which renders it useful for high-level activities, viz. taskplanning or human robot interaction.The second chapter refers to the description of the developed technologicalbackground required to build a consistent semantic map. Therefore, a majorcontribution of the first part is the development of an innovative visual odometryalgorithm able to operate in real time. This algorithm receives as input successivestereo pair of images from a stereoscopic camera mounted on a mobilerobot. It involves the detection of the salient landmarks between successiveimages. A depth estimation of these features is then obtained and a novelnon-iterative outlier detection and discarding methodology able to remove boththe mismatches between the features and the inserted errors due to the 3Dreconstruction procedure. A hierarchical motion estimation technique, whichproduces robust estimations for the movement of the robot is then adopted,thus providing refinements to the robot’s global position and orientation. Anadditional 3D reconstruction algorithm that operates on stereo images hasbeen developed providing accurate reconstruction of the area observed bythe robot. Moreover, the localization algorithm comprises the cornerstone forthe development of a simultaneously localization and mapping system suitablefor the 3D geometrical mapping of the explored environment. This 3D metricmapping system is based solely on an RGB-D sensor, where in course of robot’slocomotion 3D point clouds are merged with respect to the visual odometry.The resulted 3D map is refined by exploiting a random sample consensus planedetection algorithm accompanied by an iterative closest point registration step,among the dominant planes of the consecutive time instances, resulting thusin a very consistent geometrical 3D map. All the aforementioned developedalgorithms have been evaluated on a custom made robot platform bearing twostereoscopic cameras with different baselines and a RGB-D sensor.The third chapter encloses all the semantic mapping methods based onvisual cues that have been developed within this PhD dissertation. The firstone examines the overall traversability of the observed scene taking into considerationthe robot’s embodiment. This knowledge constitutes a cornerstone forthe autonomous robot navigation. The developed system utilizes an algorithm to retrieve specific characteristics of the environment using a stereo cameraand to produce a disparity map of the scene. Then, the v-disparity image iscalculated based on the disparity map. The v-disparity is then exploited by afeature extraction procedure to provide the system with respective conceptualvectors, which are used to train support vector machines in order to assess theoverall traversability of the scene. The traversable classified scenes are furtherprocessed and the likelihood distribution of the collision risk assessment, whenthe robot moves towards any direction, is calculated. The second part involvesthe description of a novel object recognition algorithm based on hierarchical temporalmemory networks. This constitutes a supervised learning method usedto recognize objects in different orientations. It introduces specific alternativerules for the design of each building block of a hierarchical temporal memorynetwork. These rules expand both the spatial and the temporal module ofthe network. Various type of input layers have been tested such as logppolarand saliency detection ones in order to find the solution that fits better in applicationsthat concern cluttered environments. The third part of this chaptercomprises the description of an innovative place classification algorithm. Withinthis work, in course of robot’s locomotion, salient visual features are detectedand they shape a bag-of-features problem, quantized by a neural gas to codethe spatial information for each scene. Each input image is transformed intoan appearance based histogram representation that abstr