TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation
Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, Dongwon Lee · 2021
Recent progress in generative language models has enabled machines to generate astonishingly realistic texts.While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection).However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods.In this work, we present the TURINGBENCH benchmark environment, which is comprised of (1) a dataset with 200K human-or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANS-FORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks-i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards.Our preliminary experimental results using TURINGBENCH show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models.The TURINGBENCH is available at: https: //turingbench.ist.