Random Forest Framework Customized to Handle Highly Correlated Variables: An Extensive Experimental Study Applied to Feature Selection in Genetic Data
Christine Sinoquet, Kamel Mekhnacha · 2018
The random forest model is a popular framework used in classification and regression. In cases where high correlations exist within the data, it may be beneficial to capture these dependencies through latent variables, for an enhanced use of the random forest framework. In this paper, we present Sylva, the second proposal of a random forest with latent variables after T-Trees, derived from the seminal works of Botta and co-workers (Botta et al., 2008). Sylva is an innovative hybrid approach in which the dynamic generation of latent variables used to learn the random forest is driven by an additional forest model, this time a forest of latent tree models. The latter forest model, a class of Bayesian networks devised in (Mourad et al., 2011), allows a flexible modeling of the dependencies existing within the data. In the comprehensive study reported here, three variants of Sylva, instantiated by different clustering methods (CAST, DBSCAN, Louvain method), are compared to T-Trees using high-dimensional real-world datasets (161 datasets each describing around 5,000 observations and between 5,700 and 39,000 variables) in the context of genetic association studies. We show that T-Trees and Sylva have comparable high predictive powers (aeras under the ROC curves), that lie in range [0.887, 0.961] (T-Trees), and in interval [0.885, 0.979] (over the three Sylva instantiations). Interestingly, T-Trees and Sylva are shown to differ significantly in their importance measure distributions: in Sylva, the importance measure distribution corresponding to top ranked variables is significantly skewed towards higher values than in T-Trees, which meets the feature selection enhancement objective. This property holds true for the three instantiations of Sylva. In addition, the thorough analysis of the number of top-ranked variables jointly identified by T-Trees and Sylva highlights the possibility to cross-validate the findings, in order to constitute a priorized list of features (e.g., to be further analyzed by biologists, in the context of genetic association studies). Finally, we conclude that it is recommended to use CAST or DBSCAN, and not the Louvain method, on the 161 datasets analyzed, to increase the probability of Sylva to detect top variables missedby T-Trees among its top ranked variables.