Hierarchical Topic Detection in TDT-2004
Ao Feng, James W Allan · 2004
Huge volume of news makes it hard for people to keep up with the latest information, and automatic processing of news information becomes necessary. Topic Detection and Tracking is a research program that deals with this problem. From the observations in TDT, news topics can be described in different sizes, making it hard to define the “correct ” granularity. In TDT-2004, the topic detection task was replaced by a new task called hierarchical topic detection, which used a hierarchy to capture more possible granularities. This paper shows the task definition, evaluation schemes, our attempt to generate a proper hierarchy, comparisons of different participants, and detailed analysis of the results. In the hierarchical structure, not all units satisfy the definition of topics. Our assumption is that the units between topics and stories are events, or sub-topics, and dependency analysis among events can give us better understanding of topics and other concepts. We propose a formal framework of event dependencies, but it still needs data collections, evaluation schemes and actual experiments to test its validity.