Frame-Level Multiple Sound Sources Localization Based on Visual Understanding
Lixuezi Fang, Long Ye, Xin Ma, Ruiqi Wang, Zhong Wei, Qin Zhang · 2021 5th Asian Conference on Artificial Intelligence Technology (ACAIT) · 2021
Sound source localization is an important field of audio and visual research. In the dynamic performance stage, finding the positions of multiple sounding objects in real time can give the audience an immersive feeling. Due to the complexity of the performance scene, it is a challenge to perform audio-visual recognition and localization because of the audio overlapping and visual object masking. To address this problem, we propose a novel two-stream learning framework that disentangles different classes of audio-visual representations from complex scenes, then maps the audio area of each visual in multi-instance labels learning through adaptive multi-stream fusion, and localizes sounding instrument from coarse to fine. We have obtained the state-of-the-art results on the public dataset. Experiment results show that our method can effectively realize frame-level multiple sound sources location.