Structured Topic Models for Language
Hanna Wallach · 2008
This thesis introduces new methods for statistically modelling text using topic models. Topic models have seen many successes in recent years, and are used in a variety of applications, including analysis of news articles, topic-based search interfaces and navigation tools for digital libraries. Despite these recent successes, the field of topic modelling is still relatively new and there remains much to be explored. One noticeable absence from most of the previous work on topic modelling is consideration of language and document structure—from low-level structures, including word order and syntax, to higher-level structures, such as relationships between documents. The focus of this thesis is therefore structured topic models—models that combine latent topics with information about document structure, ranging from local sentence structure to inter-document relationships. These models draw on techniques from Bayesian statistics, including hierarchical Dirichlet distributions and processes, Pitman-Yor processes, and Markov chain Monte Carlo methods. Several methods for estimating the parameters of Dirichlet-multinomial distributions are also compared. The main contribution of this thesis is the introduction of three structured topic models. The first is a topic-based language model. This model captures both word order and latent topics by extending a Bayesian topic model to incorporate n-gram statistics. A bigram version of the new model does better at predicting future words than either a topic model or a trigram language model. It also provides interpretable topics. The second model arises from a Bayesian reinterpretation of a classic generative dependency parsing model. The new model demonstrates that parsing performance can be substantially improved by a careful choice of prior and by sampling hyperparameters. Additionally, the generative nature of the model facilitates the inclusion of latent state variables, which act as specialised part-of-speech tags or “syntactic topics”. The third is a model that captures high-level relationships between documents. This model uses nonparametric Bayesian priors and Markov chain Monte Carlo methods to infer topic-based document clusters. The model assigns a higher probability to unseen test documents than either a clustering model without topics or a Bayesian topic model without document clusters. The model can be extended to incorporate author information, resulting in finer-grained clusters and better predictive performance.