Modeling the statistical behavior of lexical chains to capture word cohesiveness for automatic story segmentation
Shing-kai Chan, Lei Xie, Helen M. L. Meng · 2007
We present a mathematically rigorous framework for mod-eling the statistical behavior of lexical chains for automatic story segmentation of broadcast news audio. Lexical chains were first proposed in [1] to connect related terms within a story, as an embodiment of lexical cohesion. The vocabulary within a story tends to be cohesive, while a change in the vocabulary dis-tribution tends to signify a topic shift that occurs across a story boundary. Previous work focused on the concept and nature of lexical chains but performed story segmentation based on ar-bitrary thresholding. This work proposes the use of the log-normal distribution to capture the statistical behavior of lexical chains, together with data-driven parameter selection for lexical chain formation. Experimentation based on the TDT-2 Man-darin Corpus shows that the proposed statistical model leads to better story segmentation, where the F1-measure increased from 0.468 to 0.641. Index Terms: story segmentation, spoken document retrieval, Chinese