Hybrid Convolutional Neural Network-Transformer Model for End-to-End Binaural Sound Source Localization in Reverberant Environments
Xinyi Chen, Lijia Zhao, Jie Cui, Hua Li, Xiaodong Wang · IEEE Access · 2025
The end-to-end binaural sound source localization model can implicitly extract features from the original signal waveforms and take full advantage of neural networks. In this study, we propose a new deep learning-based end-to-end binaural sound source localization model called WavLocT, which employs a gammatone filter bank to simulate the frequency properties of the human ear and decomposes the signals received by microphones into subband signals. The WavLocT model has a unique feature extraction block that incorporates a convolutional neural network (CNN) and transformer structure, in which binaural localization features are extracted from different subband signal waveforms via the CNN structure. Additionally, we use a selective attention mechanism for different frequency subbands via the transformer encoder. In the training phase, we simulated three reverberant rooms of different sizes and trained two models separately for diffuse and directional noise environments. In the testing phase, we applied the trained models to both binaural room impulse response (BRIR)-matched and BRIR-mismatched environments. We selected the root mean square error (RMSE) and accuracy (Acc) as evaluation metrics to evaluate model performance and compared the results with those of two recent CNN-based end-to-end binaural sound source localization models. The results of the simulation experiments demonstrated that the proposed WavLocT model effectively estimated the azimuth of the desired speech signal in both diffuse and directional noise environments. Specifically, in the diffuse noise environment, WavLocT achieved an average RMSE of 6.45° and Acc of 70.02% across all mismatched rooms, outperforming WavLoc (6.72° RMSE, 56.10% Acc) and WavLocEC (7.46° RMSE, 65.86% Acc). In the directional noise environment, WavLocT achieved an average RMSE of 7.61° and Acc of 63.38% across all mismatched rooms, outperforming WavLoc (8.34° RMSE, 52.38% Acc) and WavLocEC (8.19° RMSE, 58.05% Acc).