Closing the Gap Between Single-User and Multi-User VoiceFilter-Lite
Rajeev Vijay Rikhye, Quan Wang, Qiao Liang, Yanzhang He, Ian McGraw · 2022
VoiceFilter-Lite is a speaker-conditioned voice separation model that plays a crucial role in improving speech recognition and speaker verification by suppressing overlapping speech from non-target speakers.However, one limitation of VoiceFilter-Lite, and other speaker-conditioned speech models in general, is that these models are usually limited to a single target speaker.This is undesirable as most smart home devices now support multiple enrolled users.In order to extend the benefits of personalization to multiple users, we previously developed an attention-based speaker selection mechanism and applied it to VoiceFilter-Lite.However, the original multi-user VoiceFilter-Lite model suffers from significant performance degradation compared with single-user models.In this paper, we devised a series of experiments to improve the multi-user VoiceFilter-Lite model.By incorporating a dual learning rate schedule and by using feature-wise linear modulation (FiLM) to condition the model with the attended speaker embedding, we successfully closed the performance gap between multi-user and single-user VoiceFilter-Lite models on single-speaker evaluations.At the same time, the new model can also be easily extended to support any number of users, and significantly outperforms our previously published model on multi-speaker evaluations.