Toward Better Storylines with Sentence-Level Language Models
Daphne Ippolito, David Grangier, Douglas Eck, Chris Callison-Burch · 2020
We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives.Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multisentence coherence.Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embeddings and predicts an embedding for the next sentence, which is more efficient than predicting word embeddings.Notably this allows us to consider a large number of candidates for the next sentence during training.We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks.