Developing a ChatGPT-Based Text Extraction Model to Analyze Effective Communication Elements in Pandemic-Related Social Q&A Responses
Hyunwoo Moon, Beom Jun Bae, Sangwon Bae · 2024
The present study attempts to use the large language model (LLM) to create a model that identifies Aristotle's rhetorical principles - ethos (source credibility), pathos (emotional appeal), and logos (logic) - in response to COVID-19 information on a question-and-answer community (social Q&A platform). The model differentiates between the most upvoted and random answers to analyze the presence of subdimensions of these rhetorical principles. The research utilized answers to COVID-19 questions on Naver Knowledge-iN, the most popular social Q&A platform in South Korea. A set of 193 answer pairs was randomly selected for training (135 pairs) and testing (58 pairs). These answers were coded for the three rhetorical principles and their subdimensions by researchers, which were used to refine models based on GPT 3.5 technology. The F1 scores were improved to. 88 (ethos),. 81 (pathos), and. 69 (logos). The fine-tuned models were employed to analyze 128 newly drawn answer pairs of the most upvoted answers and random answers. The paired sample t-tests indicated that rhetorical elements of logos such as factual information and logical reasoning were positively associated with health consumers' preference of information (answers) while the other rhetorical principles of ethos and pathos were not associated with consumer preference of health information. By utilizing the LLM for the analysis of persuasive content, which has been typically conducted manually with much labor and time, this study not only demonstrates feasibility of using the LLM in studies of the humanities and social sciences, but also contributes to expanding the horizon in the field of AI text extraction.