Seeing the Invisible: Recovering Surveillance Video With COTS mmWave Radar
Mingda Han, Huanqi Yang, Mingda Jia, Weitao Xu, Yanni Yang, Zhijian Huang, Jun Luo, Xiuzhen Cheng, Pengfei Hu · IEEE Transactions on Mobile Computing · 2024
Video surveillance systems play a crucial role in ensuring public safety and security by capturing and monitoring critical events in various areas. However, traditional surveillance cameras face limitations when it comes to malicious physical damage or obscuring by offenders. To overcome this limitation, we proposem$^{2}$2Vision, which is the first millimeter-wave (mmWave)-based video reconstruction system designed to enhance existing video surveillance cameras.m$^{2}$2Visionutilizes mmWave to sense the profile and motion signature of the target, integrating it with previously acquired visual data about the environment and the target's appearance, thereby facilitating the reconstruction of surveillance video. Specifically, our proposed system incorporates a dual-stage mmWave signal denoising algorithm to efficiently eliminate the noise and multiple-input multiple-output virtual antenna enhanced heatmap generation (MVAE-HG) method to obtain fine-grained mmWave heatmaps responsive to the target's profile and motion information. Moreover, we design the mm2Video generative network that first employs a multi-modal fusion module to fuse the mmWave and pre-acquired visual data, then use a conditional generative adversarial network (cGAN)-based video reconstruction module for surveillance video reconstruction. We conducted comprehensive experiments onm$^{2}$2Visionusing a commercial mmWave radar and four surveillance cameras across various environments, with the participation of seven individuals. Evaluation results show thatm$^{2}$2Visioncan achieve an average structural similarity index measure (SSIM) of 0.93, demonstrating its effectiveness and potential.