Evaluating Unsupervised Hierarchical Topic Models Using a Labeled Dataset
ULiege/HEC, Liege, Belgium, Judicaël Poumay, Ashwin Ittoo, ULiege/HEC, Liege, Belgium · 2023
Topic models are often evaluated with measures such as perplexity and topic coherence.However, these methods fall short in determining the comprehensiveness of identified topics.This research introduces a complementary approach to evaluating unsupervised topic models using a labeled dataset.By training hierarchical topic models and utilizing known labels for evaluation, we found a high accuracy of 70% for expected topics.Despite having 90 labels in the dataset, even those representing only 1% of the data achieved an average accuracy of 37.9%, illustrating hierarchical topic models' effectiveness on smaller subsets.Additionally, we confirmed that this new evaluation method helps assess the topic tree quality, demonstrating that hierarchical topic models generate coherent taxonomies.Lastly, we established that coherence measures alone are insufficient for a holistic topic model evaluation.