Selfie Segmentation in Video Using N-Frames Ensemble
Yong‐Woon Kim, Yung-Cheol Byun, Addepalli V. N. Krishna, Karthika K Balachandran · IEEE Access · 2021
Many camera apps and online video conference solutions support instant selfie segmentation or virtual background function for entertainment, aesthetic, privacy, and security reasons. A good number of studies show that Deep-Learning based segmentation model (DSM) is a reasonable choice for selfie segmentation, and the ensemble of multiple DSMs can improve the precision of the segmentation result. However, it is not fit well when we apply these approaches directly to the image segmentation in a video. This paper proposes an N-Frames (NF) ensemble approach for a selfie segmentation in a video using an ensemble of multiple DSMs to achieve a high-performance automatic segmentation. The proposed NF ensemble approach executes only one segmentation model upon a current video frame and combines segmentation results of previous frames to produce the final result. For the experiment, we use four state-of-the-art image segmentation models and 81 videos dataset with a single-person view from publicly available websites. This paper calculates Intersection over Union (IoU), IoU standard deviation, false prediction rate, Memory Efficiency Rate and Computing power Efficiency Rate to measure the performance of segmentation models. The average IoU value of the Two-Frames NF ensemble was 95.1253%, and the Three-Frames NF ensemble was 95.1734%, whereas the average IoU value of single models was 92.9653%. The result shows that the proposed approach improves the accuracy of selfie segmentation by more than 2% on average. The result of cost efficiency measurement shows that the proposed method consumes less computing power like single models. To sum up the overall work, the proposed approach is as fast as single selfie segmentation models. At the same time, it produces optimized and improved segmentation results like the ensemble of multiple segmentation models at once.