Evaluation of Change Detection Algorithms using Difficulty Maps
Sílvio Ricardo Rodrigues Sanches, Cléber Gimenez Corrêa, Beatriz Regina Brum, Pedro H. Bugatti, Priscila T. M. Saito, Claudinei Moreira da Silva, Elton Custódio · IEEE Latin America Transactions · 2023
The evaluation of a change detection algorithm should show its superiority over state-of-the-art algorithms' performances. Evaluating an algorithm involves executing it to segment a set of videos and comparing the results with the ground truth. Here, we used the difficulty level to classify each pixel of each frame of the videos of a dataset as an algorithm performance measure. A structure called "difficulty map" stores information about the difficulty of classifying each pixel in a frame. Based on these maps, we developed a metric that aims to evaluate the performance of algorithms on the difficulty map. The results showed that there are algorithms with the characteristic of classifying pixels that most state-of-the-art algorithms cannot classify (promising algorithms). Identifying such algorithms is essential since improving their performance means facing challenges already overcome by existing approaches.