Blink-former: Light-aided beamforming for multiple targets enhancement
Daiki Horiike, Robin Scheibler, Yukoh Wakabayashi, Nobutaka Ono · 2019
We propose a multimodal framework to enhance multiple target sound sources using a conventional microphone array, a video camera, and sound power sensors, called Blinkies, that we have recently developed. Each Blinky consists of a microphone, LEDs, a microcontroller, and a battery. One of the LEDs intensity is varied according to sound power, that is, the Blinky works as a sound-to-light conversion sensor. They are easy to distribute over a large area, and thus, the sound power information therein can be harvested by capturing the LED signals with a video camera. Although these signals are a mixture of contributions from multiple sources, we demonstrate that they can be separated into individual source activities by non-negative matrix factorization. The obtained activities are further utilized to design maximum signal-to-interference-and-noise ratio beamformers enhancing the source signals. We conduct numerical simulations and real experiments to evaluate the performance of this method in diffuse noise environment. The experimental results show that the proposed scheme using Blinkies is superior to competing algorithms, especially at low signal-to-noise ratio.