Assessing the Readability and Coherence in Gemini’s Triple Draft Generation: A Multi-metric Approach
C. P. Afsal, K. S. Kuppusamy · 2024
Large Language Models (LLMs) play a crucial role in modern society by enabling natural language understanding and generation, which enhances communication, automation, and information processing across diverse sectors, fostering innovation and efficiency. The response generated by an LLM for a given prompt is crucial as it reflects the model’s ability to understand context, convey relevant information, and maintain coherence, thereby shaping user perception and interaction. This study analyzes the coherence and readability of drafts generated by the Gemini LLM across 20 prompts. Coherence scores, assessed through semantic and cosine similarity, reveal variability across drafts, with Draft-1 generally excelling in both semantic and cosine similarity across the prompts. Readability analysis using Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) metrics indicates that Drafts 2 and 3 consistently offer higher readability compared to Draft 1. However, user interaction with Gemini’s interface reveals the potential overlook of alternative drafts, highlighting the importance of exploring all options for optimal content selection. This study underscores the significance of considering coherence and readability metrics collectively for informed decision-making in content-generation tasks.