A RETRIEVAL-AUGMENTED GENERATION FRAMEWORK FOR EXPLAINABLE ACADEMIC PAPER QUALITY ASSESSMENT
WeiJing Zhu, RunTao Ren, Wei Xie, CenYing Yang · Eurasia Journal of Science and Technology · 2025
With the exponential growth of global scholarly output, traditional academic paper evaluation methods face significant challenges in reliability, consistency, and scalability. Peer review processes suffer from low inter-rater agreement and lengthy decision times, while bibliometric approaches systematically disadvantage emerging fields. To address these systemic limitations, this study proposes a novel evaluation framework leveraging Retrieval-Augmented Generation (RAG) architecture and large language models (LLMs). The framework implements a four-dimensional assessment mechanism—analyzing research questions, methodologies, results, and conclusions—supported by contextual knowledge retrieval and explainable judgment generation. Experimental validation demonstrates the superiority of the RAG-based approach over both human experts and conventional machine learning baselines, achieving an F1-score of 0.77 at the quartile level. Additionally, the system provides transparent evaluative judgments supported by comparable evidence from prior literature. This work contributes to advancing scholarly communication by offering a scalable, explainable, and reliable alternative to existing evaluation paradigms.