Subtopic Division in News Topic Based on Latent Dirichlet Allocation
yan zheng · Journal of Chinese Computer Systems · 2013
Nowadays it is difficult to distinguish the subtopics in a hot news topic on the internet.To solve this problem,in the paper,the method of subtopic division based on Latent Dirichlet Allocation is presented.It describes a news document by Latent Dirichlet Allocation,and uses Bayes standard method to determine the optimal number of topics in order to fit documents best.According to the high similarity of documents between subtopics,the relativity analysis of feature words is introduced.Using the improved Kullback-Leibler distance to calculate the similarity of news stories can distinguish the stories which have similar content but belong to different topics effectively.Finally,it divides a hot news topic to subtopics by clustering the news documents with the single-pass incremental clustering algorithm.Experimental results verify the availability of the improved similarity calculation method,and it shows that this method can improve the performance of subtopic division effectively comparing to the baseline method.