Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models
Avery Hiebert, Cole Peterson, Alona Fyshe, Nishant A. Mehta · 2018
While Long Short-Term Memory networks (LSTMs) and other forms of recurrent neural network have been successfully applied to language modeling on a character level, the hidden state dynamics of these models can be difficult to interpret.We investigate the hidden states of such a model by using the HDB-SCAN clustering algorithm to identify points in the text at which the hidden state is similar.Focusing on whitespace characters prior to the beginning of a word reveals interpretable clusters that offer insight into how the LSTM may combine contextual and character-level information to identify parts of speech.We also introduce a method for deriving word vectors from the hidden state representation in order to investigate the word-level knowledge of the model.These word vectors encode meaningful semantic information even for words that appear only once in the training text. Cluster Sample Members of Cluster POS -Precision POS -RecallHS-35 asked; replied; remarked; continued; replied;