Single channel speech denoising by DDPG reinforcement learning agent
Sania Gul, Muhammad Salman Khan · Applied Acoustics · 2025
• A novel speech denoising algorithm is presented based on the deep deterministic policy gradient (DDPG) agent; an off-policy reinforcement learning (RL) agent with a continuous action space. • Two separate DDPG agents are trained on the magnitude and phase components of the short-time Fourier transform (STFT) of the speech signal. • After training, the DDPG agents generate the magnitude and phase soft masks, when noisy speech is given as input to them. These masks are then applied to the complex STFT matrix of the noisy speech to obtain the denoised speech. • Unlike the earlier RL-based speech enhancement models, our proposed model is a completely autonomous speech denoising model in its own right. However, it can also be cascaded with other state-of-the-art models to improve their performance. • As an autonomous system, it offers the highest perceptual evaluation of speech quality (PESQ) over the unprocessed speech as compared to other baseline systems by utilizing data that is 7 times smaller than that required by different models and 12 times lesser learnable parameters than the next most compact model compared here. Speech denoising (SD) covers the algorithms that suppress the background noise from the contaminated speech and improve its clarity. In this paper, a novel SD algorithm is presented based on the deep deterministic policy gradient (DDPG) agent; an off-policy reinforcement learning (RL) agent with a continuous action space. The noisy speech is first converted from the time domain to the time – frequency (TF) domain by taking its short-time Fourier transform (STFT), and then two separate DDPG agents are trained on the magnitude and phase components of the STFT. The reward function used for training these agents is the relative perceptual quality score of speech. After training, the DDPG agents generate the magnitude and phase soft masks, when noisy speech is given as input to them. These masks are then applied to the complex STFT matrix of the noisy speech to obtain the denoised speech. For matched testing data, the proposed system offers an improvement of 1.55 points in the perceptual evaluation of speech quality (PESQ) over the unprocessed speech, the highest among the other recent state-of-the-art models used for comparison in this paper. It achieves this performance by utilizing data that is 7 times smaller than that required by other models. Also, its learnable parameters are the lowest among all models, almost 12 times less than the next most compact model, based on another continuous RL agent (policy gradient (PG)) for estimating its convolutional kernels. When cascaded with a coloured spectrogram-based SD model, the proposed model further improves the PESQ by 0.07, CSIG by 1.23, and COVL by 1.4 points; the metrics estimating respectively the perceived quality of speech, its composite distortion, and its composite overall quality, surpassing all other baseline systems compared here. In the cascaded configuration, our proposed model offers the highest gain in PESQ (by 0.78 points) at 50 times fewer episodes, compared to the already trained speech enhancement and recognition models utilizing discrete agents for their performance improvement.