From Zero-Shot to Iterative Prompting: Evaluating LLMs in Automated Abstract Generation
Alexey Dmitrievich Timokhov · Litera · 2025
This study evaluates the performance and nuances of several large language models — ChatGPT, Gemini, Mistral, and Llama — focusing on their capacity to generate abstracts for academic articles in Russian and English. The analysis explores how these models perform in terms of linguistic quality and cross-linguistic adaptation, as well as their adherence to established conventions of different academic traditions. The study also evaluates both the overall performance and the practical relevance of different prompting approaches, focusing on zero-shot and iterative prompting. Drawing on a corpus of 100 academic articles published between 2018 and 2023 across humanities and technical fields, the research examines the ability of these models to handle a wide spectrum of subject matter and genre-specific demands. Special attention is given to identifying differences between models, both in terms of stylistic and structural preferences and in the context of cross-linguistic adaptation when generating titles in Russian and English. The research applies unified zero-shot prompts based on concise summaries of the original articles, followed by iterative prompting aimed at improving initial outputs by addressing identified shortcomings. The generated abstracts are evaluated according to structural coherence, terminological accuracy, stylistic conformity, completeness, and informational relevance. The findings indicate that all tested models are generally capable of producing abstracts consistent with essential genre-specific and stylistic conventions, yet each model exhibits distinctive generation strategies. While zero-shot prompting typically produces acceptable results in both languages, it often causes issues related to insufficient detail, accuracy, and adherence to stylistic norms in Russian. Conversely, iterative prompting significantly improves abstract quality by addressing these specific shortcomings. This paper offers the first comparative multilingual analysis of several large language models within the context of academic discourse, introducing linguistic community and academia to an emerging type of research material — AI-generated texts, as opposed to conventionally authored texts produced directly by humans. The findings confirm that large language models, given proper guidance, can generate academic abstracts comparable in quality to those written by humans, though their use requires careful editorial oversight to ensure full compliance with academic standards.