Comparing Character-level Neural Language Models Using a Lexical Decision Task
Gaël Le Godais, Tal Linzen, Emmanuel Dupoux · 2017
What is the information captured by neural network models of language?We address this question in the case of character-level recurrent neural language models.These models do not have explicit word representations; do they acquire implicit ones?We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word.We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width.We also compare these architectural properties to a simple count of the parameters of the network.The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.