Response to “Incorporating Biological, Chemical, and Toxicological Knowledge Into Predictive Models of Toxicity”
Russell S. Thomas, Michael B. Black, Lili Li, Eric Healy, Tzu‐Ming Chu, Wenjun Bao, Melvin E. Andersen, Russell D. Wolfinger · Toxicological Sciences · 2012
We thank Dix and colleagues for taking the time to consider and respond to our recent article on predicting in vivo hazard using high-throughput in vitro screening (Thomas et al., 2012). To distill down the most important point of their letter, they state that our study involves a “statistical path to producing classifiers that does not incorporate knowledge of biological or adverse outcome pathways to group assays or endpoints from the in vitro or in vivo datasets.” We disagree with this conclusion. The underlying issue is not about our failure to incorporate biological knowledge; it revolves around the different ways in which the feature selection process was performed in our study compared with their studies. The feature selection process is the way that features (i.e., in vitro assays) are selected for inclusion into the machine-learning model. As stated by Dix and colleagues, the process of feature selection generally improves the performance of machine-learning models by removing irrelevant features. However, removal of these irrelevant features must be done in a statistically valid way in order to avoid biasing the predictive performance of the model. We will respond to their concerns and clarifications by first comparing their predictive modeling approach with ours and then discuss the overall merits of their approach in the context of predicting in vivo hazard. Dix and colleagues list a total of seven publications where they apply their predictive modeling approach. For the sake of brevity, we focus here on two of their publications (Martin et al., 2011; Sipes et al., 2011) and compare their approach with ours. In both the Martin et al. and Sipes et al. publications, the feature selection process was performed outside the cross-validation loop using a series of univariate statistical tests (i.e., t-test, correlation and chi-square test). The univariate statistical tests were performed using the output from each individual in vitro assay (i.e., log-transformed AC50 or LEC values) and comparing the response in the in vitro assay with either continuous or discretized values from the in vivo endpoint being predicted across all chemicals. The in vitro assays that were significantly associated with the in vivo endpoint were then aggregated based on gene, gene family, or gene ontology biological process. In the Sipes et al. study, a second level of aggregation was performed that maximized the univariate statistical association (Fig. 1 in Sipes et al., 2011). The aggregation was also performed outside of the cross-validation loop. Notably, Martin et al. did not adhere to a biologically based form for aggregation of the in vitro assays into genes and gene families. One of the final predictors in their predictive model was an aggregated variable called “Other,” which included a diverse set of four in vitro assays with no common biological function. In our study, feature selection was performed inside the cross-validation loop and also employed a univariate statistical test (i.e., t-test). The in vitro assays were also aggregated based on gene and gene ontology biological process. This aggregation was performed outside the cross-validation loop but included all in vitro assays (i.e., not just those that passed the univariate statistical tests) in order to avoid introducing bias into the predictive performance. The use of feature selection outside versus inside the cross-validation loop is an important difference between our two approaches. Proper statistical methodology dictates that feature selection should be performed inside the cross-validation loop to avoid contaminating assay selection and biasing estimates of model performance (Molinaro et al., 2005; Shi et al., 2010). When performed outside the cross-validation loop, essentially all chemicals inform the selection of the in vitro assays, a process that negates the use of cross-validation as an objective way to evaluate performance. If feature selection is performed outside the cross-validation loop, then a properly selected external validation set is needed to objectively evaluate predictive performance. The Sipes et al. study did not use an external validation set. In the Martin et al. study, the external validation set consisted of chemicals that did meet their own exclusion criteria for model development (Tables 2 and 5 in Martin et al., 2011). All external validation chemicals in the Martin et al. study had either unacceptable or entirely lacked guideline reproductive studies. The in vivo responses for the chemicals were instead obtained from nonstandard studies in the open literature. Further, a subset of the external validation chemicals in the Martin et al. study had little or no activity in the in vitro high-throughput screening assays, a feature that was also grounds for exclusion during model development based on their own criteria. In our opinion, these chemicals do not conform to a properly selected external validation set. Even if the 21 external chemicals were representative, results for a single external hold out set provide no standard error, so we do not have any estimate of uncertainty attached with their estimate of balanced accuracy (76%). In our study, the prediction of rat reproductive toxicity (M_Rat_Reproduct_Outcome) had a median balanced accuracy of 52.5±1.4% (SD) across the 84 different statistical classification models. In both the Martin et al. and Sipes et al. publications, linear discriminant analysis was used as the machine-learning algorithm with fivefold cross-validation. In our study, linear discriminant analysis was also used as one of the eight different machine-learning algorithms. Our rationale for using a multiple machine-learning algorithms was not to achieve overfitting as Dix and colleagues imply but to broadly assess the predictive performance of the in vitro assays while remaining agnostic to the specific machine-learning approach. If overfitting was achieved in our study, then our estimates of the predictive performance of the in vitro assays would have been significantly higher. Finally, our study also employed fivefold cross-validation as a way to evaluate predictive performance. In our view, the integrated feature selection methods employed by the Martin et al. and Sipes et al. studies represent statistical data mining approaches with some attempt to aggregate predictor variables based on similar biological targets (i.e., genes) or higher order biological function (i.e., gene ontology biological processes). These data mining approaches have a long history in the microarray field for interpreting data and inferring functional significance. We certainly have no objection to their use of statistical data mining approaches to identify those in vitro assays that are associated with in vivo responses or the attempt to place these assays into biological or adverse outcome pathways. What we strongly object to is the coupl ing of these statistical data mining approaches with machine-learning algorithms using a biased statistical methodology while presenting the approach under the guise of “hypothesis-driven” and “incorporating biological, chemical, and toxicological knowledge.” It is also contradictory that Dix and colleagues criticized our study as “a statistical path to producing classifiers that does not incorporate knowledge of biological or adverse outcome pathways to group assays or endpoints from the in vitro or in vivo datasets.” We actually employed one of the same statistical tests for feature selection that they employed, included the same machine-learning algorithm as they used, and also aggregated the in vitro assays based on genes and biological processes. The only difference in our approaches was that the feature selection and assay aggregation were performed in a statistically valid manner that did not bias estimates of predictive performance. For readers in the toxicology community with a sophisticated understanding of machine-learning methods, this may not present an issue as they understand the implications and limitations of the analysis performed by the ToxCast group. However, readers without this knowledge who accept the results at face value may be led to believe that the in vitro assays have greater predictive performance than they really do. Finally, we do commend the ToxCast program for their efforts in compiling and organizing the in vivo toxicity data in a standardized, computable format and for the effort involved in performing and analyzing the in vitro assay data. We fully appreciate the work involved in data collection, database construction, and data analysis. We also agree with Dix and colleagues that this information will eventually help make better decisions on chemicals by identifying potential molecular initiating events and developing a mode-of-action understanding for the properties of specific chemicals. Based on the potential of these technologies to transform the way we do toxicity testing, we look forward to the continuation of the ToxCast program and their attempts to understand associations between the high-throughput study results and apical responses. Nonetheless, the predictive modeling performed by the ToxCast group must be statistically sound, transparent, and avoid misrepresenting the ability to predict chemical hazard. We stand by our initial conclusions that the current ToxCast phase I assays and chemicals have limited ability to predict in vivo hazard.