Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Jason Lucas, Adaku Uchendu, Michiharu Yamashita, Jooyoung Lee, Shaurya Rohatgi, Dongwon Lee · 2023
Recent ubiquity and disruptive impacts of large language models (LLMs) have raised concerns about their potential to be misused (i.e., generating large-scale harmful and misleading content).To combat this emerging risk of LLMs, we propose a novel "Fighting Fire with Fire" (F3) strategy that harnesses modern LLMs' generative and emergent reasoning capabilities to counter human-written and LLM-generated disinformation.First, we leverage GPT-3.5-turbo to synthesize authentic and deceptive LLMgenerated content through paraphrase-based and perturbation-based prefix-style prompts, respectively.Second, we apply zero-shot incontext semantic reasoning techniques with cloze-style prompts to discern genuine from deceptive posts & news articles.In our extensive experiments, we observe GPT-3.5-turbo'szero-shot superiority for both in-distribution and out-of-distribution datasets, where GPT-3.5-turboconsistently achieved accuracy at 68-72%, unlike the decline observed in previous customized and fine-tuned disinformation detectors.Our codebase and dataset are available at https://github.com/mickeymst/F3.