Keep CALM and Explore: Language Models for Action Generation in Text-based Games
Shunyu Yao, Rohan Rao, Matthew Hausknecht, Karthik Narasimhan · 2020
Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces.In this paper, we propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state.Our key insight is to train language models on human gameplay, where people demonstrate linguistic priors and a general game sense for promising actions conditioned on game history.We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize ingame rewards.We evaluate our approach using the Jericho benchmark (Hausknecht et al., 2019a), on games unseen by CALM during training.Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model.Surprisingly, on half of these games, CALM is competitive with or better than other models that have access to ground truth admissible actions.* * Code and data are available at https://github. com/princeton-nlp/calm-textgame.Observation: You are in the living room.There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room.You are carrying: A brass lantern . . .