SafeCultural: A Dataset for Evaluating Safety and Cultural Sensitivity in Large Language Models
Pawat Vongpradit, Aurawan Imsombut, Sarawoot Kongyoung, Chaianun Damrongrat, Sitthaa Phaholphinyo, Tanik Tanawong · 2024
The increasing use of Large Language Models (LLMs) in daily life raises important questions about ensuring their trustworthiness. While existing datasets are widely used to evaluate issues like safety and hallucinations, they often overlook important factors like social norms and cultural sensitivity. This paper introduces a dataset that incorporates regional cultural aspects, allowing LLMs to be more adaptable across diverse contexts. This dataset covers aspects of both safety and cultural sensitivity. The safety component includes seven dimensions: Unlawful Conduct, Toxicity, Violence, Privacy Violations, Harms to Minors, Adult Content, and Mental Health Issues. Cultural sensitivity is divided into three categories: Beliefs, Behaviors, and Safety. We tested across seven LLMs, the dataset aims to help models overcome cultural barriers, fostering user-friendly and culturally aware development. Furthermore, we provide guidelines for creating and evaluating culturally sensitive prompts to ensure they meet safety and cultural standards. The finding of this paper may help the development of more adaptable, culturally aware, and trustworthy LLMs for daily use.