FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models
Shramay Palta, Rachel Rudinger · 2023
It is common sense that one should prefer to eat a salad with a fork rather than with a chainsaw.However, for eating a bowl of rice, the choice between a fork and a pair of chopsticks is culturally relative.We introduce FORK, a small, manually-curated set of CommonsenseQA-style questions for probing cultural biases and assumptions present in commonsense reasoning systems, with a specific focus on food-related customs.We test several CommonsenseQA systems on FORK, and while we see high performance on questions about the US culture, the poor performance of these systems on questions about non-US cultures highlights systematic cultural biases aligned with US over non-US cultures.