An exploratory study towards applying and demystifying deep learning classification on behavioral big data
Sofie De Cnudde, David Martens, Foster John Provost · RePEc: Research Papers in Economics · 2018
The superior performance of deep learning algorithms in fields such as computer vision and natural language processing has fueled an increased interest towards these algorithms in both research and in practice. Ever since, many studies have applied these algorithms to other machine learning contexts with other types of data in the hope of achieving comparable superior performance. This study departs from the latter motivation and investigates the application of deep learning classification techniques on big behavioral data while comparing its predictive performance with 11 widely-used shallow classifiers. In addition to the application on a new type of data and a structured comparison of its performance with commonlyused classifiers, this study attempts to shed light onto when and why deep learning techniques perform better. Regarding the specific characteristics of applying deep learning on this unique class of data, we demonstrate that an unsupervised pretraining step does not improve classification performance and that a tanh nonlinearity achieves the best predictive performance. The results from applying deep learning on 15 big behavioral data sets demonstrate as good as or better results compared to traditionally-used, shallow classifiers. However, no significant performance improvement can be recorded. Investigating when deep learning performs better, we find that worse performance is obtained for data sets with low signal-from-noise separability. In order to gain insight into why deep learning generally performs well on this type of data, we investigate the value of the distributed, hierarchical characteristic of the learning process. The neurons in the distributed representation seem to identify more nuances in the many behavioral features as compared to shallow classifiers. We demonstrate these nuances in an intuitive manner and validate them through comparison with feature engineering techniques. This is the first study to apply and validate the use of nonlinear deep learning classification on fine-grained, human-generated data while proposing efficient con guration settings for its practical implementation. As deep learning classification is often characterized by being a black-box approach, we also provide a first attempt towards the disentanglement regarding when and why these techniques perform well.