Effect of Non-linear Deep Architecture in Sequence Labeling
Mengqiu Wang, Christopher D. Manning · 2013
If we compare the widely used Conditional Random Fields (CRF) with newly proposed “deep architecture ” sequence models (Collobert et al., 2011), there are two things changing: from linear architecture to non-linear, and from discrete feature representation to distributional. It is unclear, however, what utility nonlinearity offers in conventional featurebased models. In this study, we show the close connection between CRF and “sequence model ” neural nets, and present an empirical investigation to compare their performance on two sequence labeling tasks – Named Entity Recognition and Syntactic Chunking. Our results suggest that non-linear models are highly effective in low-dimensional distributional spaces. Somewhat surprisingly, we find that a nonlinear architecture offers no benefits in a high-dimensional discrete feature space. 1