Optimizing Prompt Engineering for Automated Text Summarization of Student Reflections: A Comparative Study Using GPT-4 LLM
Nicole Wiktor · 2025
In the educational domain, extracting insights from student-written text has shown to be valuable for instructors. Efficiently summarizing students' reflections in a course offers instructors valuable insights to enhance students' learning experience. Therefore, quickly understanding students' impressions about the course could be very helpful to instructors for in-time and/or personalized one-on-one discussions. Achieving this often involves using natural language processing (NLP) techniques Understanding capabilities of LLMs through a series of comparative experiments involving prompt engineering is the goal of this work. We compare the summarization outputs of GPT-4 with an experimentally optimized temperature of 0.75 through a variety of experiments that include different levels of prompts, starting with base level and proceeding to increase context in the prompt. We evaluate and compare the outputs of these summaries based on a rubric from literature, evaluated by human annotators. Our findings suggest that providing more detailed context prompts help LLMs uncover less frequent and obvious student challenges and provide more detailed explanations. One notable finding showed how sensitive the LLM approach is to the distribution of the challenge types in students' reflections. In other words, all prompts regardless of their contextual details faced issues due to this misrepresenting of student challenges distributions, sometimes overstating their occurrence frequency. Therefore, further study is required to refine the data distribution impact. Despite this, the approach shows much potential to extract useful knowledge quickly. It offers valuable insights to instructors and could help in supporting students more effectively.