Gated fusion of handcrafted and deep features for robust automatic pronunciation assessment
Binghuai Lin, Liyuan Wang · 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · 2022
Automatic pronunciation assessment plays an important role in Computer-Assisted Pronunciation Training (CAPT). Traditional methods utilize handcrafted speech features such as Goodness of pronunciation (GOP), which may not provide sufficient information for the pronunciation proficiency assess-ment. The deep feature-based method suffers from overfitting due to data scarcity and large variations in pronunciation. In this paper, we propose a method for automatic pronunciation assessment by fusing both handcrafted and deep features to boost the robustness of feature representations. To better model the complementary relationship between GOP-based and deep features, we fuse the GOP-based features with deep features based on concatenation or addition with multiplicative or additive gated fusion, respectively. Experimental results based on the dataset recorded by Chinese English-as-second-Ianguage (ESL) learners and the Speechocean762 dataset demonstrate that the proposed method outperforms the previous work based on either GOP or deep features in the Pearson correlation coefficient (PCC).