Cross-Lingual Cross-Modal Consolidation for Effective Multilingual Video Corpus Moment Retrieval
Jiaheng Liu, Tan Yu, Hanyu Peng, Mingming Sun, Ping Li · Findings of the Association for Computational Linguistics: NAACL 2022 · 2022
Existing multilingual video corpus moment retrieval (mVCMR) methods are mainly based on a two-stream structure.The visual stream utilizes the visual content in the video to estimate the query-visual similarity, and the subtitle stream exploits the query-subtitle similarity.The final query-video similarity ensembles similarities from two streams.In our work, we propose a simple and effective strategy termed as Cross-lingual Cross-modal Consolidation (C 3 ) to improve mVCMR accuracy.We adopt the ensemble similarity as the teacher to guide the training of each stream, leading to a more powerful ensemble similarity.Meanwhile, we use the teacher for a specific language to guide the student for another language to exploit the complementary knowledge across languages.Extensive experiments on mTVR dataset demonstrate the effectiveness of our C 3 method.