Analysis of various topic modeling algorithms using internal-quality metrics
Astha Goyal, Indu Kashyap · 2024
Topic modeling, a conventional technique for uncovering hidden themes in textual data, has garnered substantial attention across various fields. While it is highly effective, it presents several challenges during application. The foremost challenge lies in the selection of the most suitable topic model across diverse datasets and evaluation criteria. As the field continuously evolves, there arises a necessity to objectively compare and assess the effectiveness of different topic modeling algorithms. This study aims to evaluate the performance of conventional topic modeling algorithms, with respect to various internal quality metrics. By conducting an in-depth analysis and interpretation of results using the 20-Newsgroup dataset, we analyzed various topic models, determining the optimal model. This study incorporates comprehensive range of metrics, including aspects like model coherence, diversity, classification, and significance, to assess the performance of these algorithms. The outcomes of this work provide readers deeper insights into the effectiveness of topic modeling algorithms.