Generating Suggestive Limitations from Research Articles Using LLM and Graph-Based Approach
Ibrahim Al Azher · 2024
Identifying and Generating research papers' limitations in scientific articles will enhance the transparency and rigor of scientific research. In this work, we aim to automatically generate Limitations sections in scientific articles based on other key sections, including Abstract, Introduction, Methodology, Related Work, Experiment, and Conclusion. To identify the most relevant content, we apply a cosine similarity-based approach to select important sections for input. For generating limitations, we experiment with several large language models (LLMs), including BART, T5, Pegasus, GPT-3.5, GPT-4, and Gemini. Our findings show that GPT-3.5 with Retrieval-Augmented Generation (RAG) outperforms other models in accurately generating limitations. Building on this, we plan to incorporate a graph-based model using a graph neural network (GNN) that leverages both citation networks and thematic similarity to enhance the generation of limitations. This graph model will be integrated with the LLM and RAG system. Additionally, we intend to create a strong ground truth by incorporating both OpenReview and Limitations sections in the training data. For evaluation, we will apply PointWise similarity with interpretable metrics, utilizing the LLM as a judge approach to assess outcomes, supplemented by a human-in-the-loop approach to further refine results.