Retrieval-augmented Video Encoding for Instructional Captioning
Yeonjoon Jung, Minsoo Kim, Seungtaek L. Choi, Jihyuk Kim, Minji Seo, Seung-won Hwang · 2023
Instructional videos make learning knowledge more efficient, by providing a detailed multimodal context of each procedure in instruction.A unique challenge posed by instructional videos is key-object degeneracy, where any single modality fails to sufficiently capture the key objects referred to in the procedure.For machine systems, such degeneracy can disturb the performance of a downstream task such as dense video captioning, leading to the generation of incorrect captions omitting key objects.To repair degeneracy, we propose a retrievalbased framework to augment the model representations in the presence of such key-object degeneracy.We validate the effectiveness and generalizability of our proposed framework over baselines using modalities with key-object degeneracy.