The Second Conversational Intelligence Challenge (ConvAI2)
Emily Dinan, Varvara Logacheva, Valentin Andreevich Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur D. Szlam, Iulian Vlad Serban, Lowe, Ryan, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joëlle Pineau, Mikhail Burtsev, Jason Weston · arXiv (Cornell University) · 2019
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (i) pretrained Transformer variants are currently the best performing models on this task, (ii) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations) -- in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).