Automated Detection of AI-Generated Text Using LLM Embedding-Driven ML Models
Andrei-Nicolae Vacariu, Marian Bucos, Marius Oteşteanu, Bogdan Drăgulescu · 2024
Large Language Models (LLM) have proved their ability in tasks once thought to be exclusive to humans, such as text sumarization, completion, question answering, and others. Although LLMs were present for some time, OpenAI's ChatGPT introduced them to a broader audience. Despite their ability to assist humans in various tasks, some concerns arise, as there is a big probability of misuse in areas such as fake news, plagiarism, and propaganda. Previous studies have shown that humans are unable to accurately detect generated text, which motivates the need for automated detectors. In this work, we explore whether machine learning models can distinguish between text written by humans and generated text when using embeddings as input. We process a publicly available data set, Human ChatGPT Comparison Corpus (HC3). The data set contains question-answer pairs in various domains, including open-domain, financial, medical, legal, and psychological areas. We are using Llama3, an open-source large language model, to generate the embeddings. We then evaluate the performance of four machine learning (ML) models in detecting text generated by ChatGPT with the text embeddings used as input. The ML models are the Support Vector Classifier, Naive Bayes, the K-Nearest Neighbors Classifier, and a neural network. The results show that relatively simple models can identify the generated text. SVC demonstrated the best results with an F1-score of 99.95%.