AISG's Online Safety Prize Challenge: Detecting Harmful Social Bias in Multimodal Memes
Ying Ying Lim, Ming Shan Hee, Xun Wei Yee, Yau Weng Kuan, Xinming Sim, Wesley Tay, Wee Siong Ng, See-Kiong Ng, Roy Ka-Wei Lee · 2024
Identifying internet memes that perpetuate harmful social biases is a significant challenge due to the memes' associated cultural references and multilingualism. This challenge is particularly apparent in Singapore, where multiple languages and diverse cultural backgrounds can make it more difficult to detect and address these biases. To better address this issue, the Online Safety Prize Challenge (OSPC) was held over ten weeks, focusing on the zero-shot detection of multilingual memes with harmful social bias within the Singaporean context. The OSPC featured an evaluation dataset of 1,629 memes, covering Singapore's four official languages of English, Chinese, Tamil and Malay. As Singlish is an informal, colloquial form of English that is widely used in Singapore, this challenge also included Singlish in the English dataset. The 10-week challenge attracted more than 310 participants from 34 countries, forming 135 teams. This challenge report contains the details for constructing the evaluation dataset and an overview of the systems proposed across various languages and social biases.