A Case Study on ChatGPT Question Generation
Winston Chan, Aijun An, Heidar Davoudi · 2023
The advent of transformers and the subsequent development of Large Language Models (LLMs) based on these technologies has revolutionized the field of Natural Language Processing (NLP). These models are able to understand and generate coherent natural language and hold conversations with humans continuously. Meanwhile, ChatGPT has become famous among many LLMs for its general-purpose characteristics and versatility. With that in mind, we investigate the capabilities of ChatGPT, which is very successful in many downstream NLP tasks on the task of Question Generation (QG). In particular, our experiments show that appropriate context through our designed prompts makes ChatGPT an appropriate tool for accurately performing the QG task. We compare ChatGPT’s question generation results with the state-of-the-art models, particularly on the SQuAD and car manual datasets. The results show that ChatGPT is able to compete with or even outperform some of the baseline models. Furthermore, we illustrate that we may improve ChatGPT through additional fine-tuning of the prompts. Finally, we also investigate the use of ChatGPT to evaluate QG models. While the use of ChatGPT for such purposes is still in its early stages, our results demonstrate that ChatGPT can potentially be a strong QG accuracy evaluator comparable to human evaluators.