VCSE: Time-Domain Visual-Contextual Speaker Extraction Network
Junjie Li, Meng Ge, Zexu Pan, Longbiao Wang, Jianwu Dang · Interspeech 2022 · 2022
Speaker extraction seeks to extract the target speech in a multitalker scenario given an auxiliary reference.Such reference can be auditory, i.e., a pre-recorded speech, visual, i.e., lip movements, or contextual, i.e., phonetic sequence.References in different modalities provide distinct and complementary information that could be fused to form top-down attention on the target speaker.Previous studies have introduced visual and contextual modalities in a single model.In this paper, we propose a two-stage time-domain visual-contextual speaker extraction network named VCSE, which incorporates visual and selfenrolled contextual cues stage by stage to take full advantage of every modality.In the first stage, we pre-extract a target speech with visual cues and estimate the underlying phonetic sequence.In the second stage, we refine the pre-extracted target speech with the self-enrolled contextual cues.Experimental results on the real-world Lip Reading Sentences 3 (LRS3) database demonstrate that our proposed VCSE network consistently outperforms other state-of-the-art baselines.