Intelligent Directing System for Music Concert Scene Based on Visual and Auditory Information
Caihong Wang, Chuhan Qiu, Wanxin Xu, Weijie Zhu, Hanlu Luo, L.-H. Chen, Ziyi Yu, Lin Fu, Xuanjun Chen · 2023
We propose an intelligent directing system based on visual and auditory information, which mainly focuses on the concert scene. Our system performs real-time decoding of signals recorded and transmitted during the live concert, while providing audio and video streaming data. In this way, the system will extract the physical and musical features of the audio stream. These auditory information will determine the timing of camera to be switched. As for video streaming data, we choose to conduct real-time detection of data from different cameras from different viewpoints, including the blurring degree and short scale, to determine whether there is a high "switching confidence" in the current video channel. In addition, our system also classifies the instruments received at each video channel, to match the main instrument identified from the audio information. Whenever the live performance reaches a point where a camera switching is required, combining the above factors, our system will obtain a ’switching score’ for each camera. Then the system will select the camera with the highest score for switching operation.