Can LLMs Determine Political Leaning of Polish News Articles?
Joanna Szwoch, Mateusz Staszkow, Rafał Rzepka, Kenji Araki · 2023
This study explores how Large Language Models (LLMs) can be used to analyze the political and establishment leanings of opening paragraphs in Polish online news articles that discuss controversial topics. This task, often time-consuming and costly when performed by humans, holds promise for enhancing media fairness and balance cost-effectively. It is one of the first attempts to analyze this problem in an under-resourced language like Polish. Findings reveal the challenge of task difficulty for LLMs and human annotators. Accuracy of LLMs varies, with better performance on questions related to political leaning than establishment stance. Notably, the text-davinci-002 model achieves the highest scores in wing orientation detection, whereas gpt-3.5-turbo excels in recognizing pro-/anti-government bias. This study highlights LLMs' limitations for this demanding task in an under-resourced language like Polish, emphasizing the need for alternative detection approaches.