Exploring the feasibility of incorporating cohesion in machine translation evaluation : A Coh-Metrix analysis of Google Translate, DeepL, and ChatGPT translations.
이화여자대학교, Hyoeun. Choi · 통역과 번역 · 2025
This study explores the feasibility of incorporating cohesion as an evaluation criterion beyond the sentence level inmachine translation assessment. To this end, the study employed Coh-Metrix to measure the cohesion of English translations produced by Google Translate, DeepL, and ChatGPT for 68 Korean editorials, focusing on three criteria: pronouns, connectives, and repetition. One-way ANOVA and Tukey’s HSD test were conducted to determine whether there were significant differences in cohesion among the machine translation engines. The results revealed differences in content word overlap, latent semantic cohesion between adjacent sentences, and the distribution of given and new information across translation outputs. Additionally, variations were observed in the frequency of temporal and additive connectives, as well as in the occurrence of first-person singular and plural pronouns. A detailed analysis of cases with significant differences showed that, while individual sentences may appear accurate, the translations sometimes failed to convey the original meaning accurately within a larger context, potentially hindering reader comprehension. This study is significant in that it moves beyond traditional sentence-level evaluations and suggests the potential for incorporating cohesion as a criterion in machine translation assessment.