TTIC’s Submission to WMT-SLT 23
Marcelo Sandoval-Castañeda, Yanhong Li, Bowen Shi, Diane Brentari, Karen Livescu, Gregory Shakhnarovich · 2023
We describe TTIC's submission to the WMT 2023 Sign Language Translation shared task on the Swiss-German Sign Language (DSGS) to German track.Our approach explores the advantages of using large-scale self-supervised pre-training in the task of sign language translation, over more traditional approaches that rely heavily on supervision, along with costly labels such as gloss annotations.The proposed model consists of a VideoSwin transformer for image encoding, and a T5 model adapted to receive VideoSwin features as input instead of text.On WMT-SLT 22's development set, this system achieves 2.03 BLEU score, a 59% increase over the previous best reported performance.On the official test set, our primary submission achieves 1.1 BLEU score and 17.0 chrF score.It also achieves the highest human evaluation score among all the participants.