Online learning based multiple pedestrians tracking in thermal imagery for safe driving at night
Byoung Chul Ko, Joon-Young Kwak, Jae-Yeal Nam · 2016
According to a report, night time and poor illumination driving is overall 2-3 times more dangerous then day time. For example, young people aged 18-24 were killed between 21:00 and 05:59 (the night-time and early morning) on week-days in the EU-23 countries because of road accident in 2010. As the similar pattern with EU, most pedestrian-vehicle accidents occur between 6 p.m. and 8 a.m., and the rate of pedestrian fatalities is highest between 4 a.m. and 6 a.m. in South Korea. Among several factors such as inebriated drivers and pedestrian, drowsiness, decreased visibility is the major cause of pedestrian-vehicle accident at night. To reduce the accident owing to driver's inattention at night, recent advanced driver assistance system (ADAS) has been researching on automatic pedestrian detection and tracking using night vision camera. Therefore, this tutorial focuses on introducing a multiple pedestrians tracking system using a thermal camera that is able to discern thermal energy at night-time. In a pedestrian tracking-by-detection system, multi-pedestrian detection accuracy is essential for post tracking process. Since the temperature difference between the pedestrian and background depends on the season and weather, we therefore first introduce two models for detecting pedestrians according to the season and weather, which are determined using Weber-Fechner's law. Two detection models use the optimal levels of the image scaling and search area instead of image pyramid to reduce the computational cost of image scaling for detecting multiple pedestrians of various sizes. Online learning is appropriate in the case that image frames is obtained sequentially. Theoretically offline learning could obtain global optimal solution while it is not as practical as online learning. Therefore, we introduce some state-of-the-art real-time online learning algorithms with our online learning based on boosted random ferns (BRFs) in detail based on the references as the second topic. Third, for association checking of multiple pedestrians, we explain the advantages and disadvantages of feed-forward system and global association system. Feed-forward system uses only current and past observations, which is called tracklet to estimate the current tracker's state. On contrary, global association system uses future and global information to estimate the current tracker's state in an offline step, for example, bipartite graph matching, Hungarian algorithm, dynamic programming, and min-cost max-flow network flow. Because maintaining the tracker's ID in successive frames is a challenging task owing to overlapping pedestrians in the multiple pedestrians tracking system, we introduce a few association checking algorithms to maintain the tracker's ID. As the feature for association checking, we also explain popular features in computer vision, such as the spatial proximity, velocity orientation and context (shape, size, color). Fourth, we introduce a few evaluation video sequences for pedestrian tracking such as OSli thermal pedestrian database, CVC-09 sequences, KAIST benchmark dataset, and KMliTD dataset. In particular, we are focusing on the KMliTD which contains video sequences involving a moving camera, moving multiple pedestrians, sudden shape deformations, unexpected motion changes, and partial or full occlusions between pedestrians at summer and winter night. Finally, we introduce the evaluation methods to measure the performance of the pedestrian tracking system such as Multiple Object Tracking Precision (MOTP) and Multiple Object Tracking Accuracy (MOTA), and Tracking Distance Error (TDE). The performance comparison among different tracking approaches is also presented when the proposed online tracking method is applied to benchmark data. As the further research in multiple pedestrians tracking, we will guide the fusion of sensors such as Radio Detection and Ranging (RADAR) sensor or Light Detection and Ranging (LIDAR) sensor with a camera for overcome the limitations occurred in a standalone sensor. In addition, with the increasing sensor resolutions, we mention the plan how to develop computationally feasible multi-extended object tracking algorithm.