Are LLMs Good Zero-Shot Fallacy Classifiers?
Fengjun Pan, Xiaobao Wu, Zongrui Li, Anh Tuan Luu · 2024
Fallacies are defective arguments with faulty reasoning.Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions.However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities.In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification.To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different taskspecific instructions such as extraction, summarization, and Chain-of-Thought reasoning.With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers.In general, LLMs under single-round prompting schemes have achieved acceptable zeroshot performances compared to the best fullshot baselines and can outperform them in all OOD inference scenarios and some opendomain tasks.Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs.Our analysis further underlines the future research on zero-shot fallacy classification.Codes and data are available at: https://github.com/ panFJCharlotte98/Fallacy_Detection.