Text Segmentation with Topic Models
Martin Johannes Riedl, Chris Biemann · LDV-Forum/Journal for language technology and computational linguistics · 2012
Text Segmentation with Topic ModelsThis article presents a general method to use information retrieved from the Latent Dirichlet Allocation (LDA) topic model for Text Segmentation: Using topic assignments instead of words in two well-known Text Segmentation algorithms, namely TextTiling and C99, leads to significant improvements.Further, we introduce our own algorithm called TopicTiling, which is a simplified version of TextTiling (Hearst, 1997).In our study, we evaluate and optimize parameters of LDA and TopicTiling.A further contribution to improve the segmentation accuracy is obtained through stabilizing topic assignments by using information from all LDA inference iterations.Finally, we show that TopicTiling outperforms previous Text Segmentation algorithms on two widely used datasets, while being computationally less expensive than other algorithms. JLCL 2012 -Band 27 (1) -47-69