A Deep Investigation into fastText
Jincheng Xu, Qingfeng Du · 2019
The researches on text classification range from traditional machine learning methods which depend on handcrafted features to recent deep learning models which learn complex and non-linear relationships automatically. While neural models have been proven to be effective for text classification, they operate like a black box and offer no interpretability for humans. In this paper, we study the state-of-the-art text classification model fastText and propose a mathematical approach to interpret the behavior of fastText into a score matrix. By decomposing the output of fastText into the sum of word-level importance scores, we can observe how each word contributes to the final prediction and understand the inner implementation of fastText. What is more, inspired by the idea of score matrix, we have also proposed two frequency matrix based methods which are totally obtained from simple statistics but achieve reasonable performance under certain conditions. At last, we perform extensive experiments on various benchmarking text classification datasets to offer clear explanations about how fastText makes predictions with the score matrix, and we provide a comprehensive analysis on frequency matrix based methods. The deep investigation into fastText helps us understand the decision making process as well as provides heuristic ideas on how to design more effective text classification models in the future.