Machine learning for tumor growth inhibition: Interpretable predictive models for transparency and reproducibility
Andreas Daniel Meid, Alexander Gerharz, Andreas Groll · CPT Pharmacometrics & Systems Pharmacology · 2022
Machine learning (ML) has recently enriched the possibilities for predicting clinical events using prognostic and/or predictive variables. Clinical pharmacology also benefits from more accurate but often increasingly complex ML models. For such models to be accepted in practice, it is essential to make them reproducible for the analyzing scientist and interpretable for the clinician. When reproducing an ML work, we exemplarily show how methods of interpretable ML can support to meet these prerequisites. Machine learning (ML) has become increasingly important in recent years, both in medicine and in the field of clinical pharmacology.1 This applies to early phases of drug development and complex analyses of typical clinical trial data. ML methods promise a higher predictive value, which may result from the flexible analysis of nonlinear correlations or higher-order interactions.2 At the same time, there is a desire on the clinical side that the path to good prediction should be comprehensible and transparent.3 If one now intends to reproduce such a predictive model, it is immensely important to clarify the model’s ML process and how predictions are obtained. This is exactly what we have approached when reproducing the results of the previously published original work of Chan et al.2 In particular, we addressed three fundamental questions: (1) what is needed for reproducible ML analyses, (2) how can these methods be described in a transparent and interpretable way, and (3) what else can be predicted from the original work? In the original manuscript, Chan et al. introduced a modeling platform based on the OAK study, which allows to compare different ML methods for predicting overall survival.2 In brief, the OAK study included patients with previously treated non-small cell lung cancer into a randomized, open-label, phase III trial in which they were randomly assigned to receive either atezolizumab or docetaxel once every 3 weeks.4 The source data are accessible with an adequate request to a clinical trials portal (www.vivli.org) and the original work also provides rudimentary analysis code. Random seeds are set in the analysis code, which is essential for ML methods (e.g., random forests5) in terms of reproducibility. With available software (versions) and helpful answers upon contacting the authors, many prerequisites for reproducible research seemed to be given. However, the practical situation was complicated by the fact that all baseline variables of the clinical trial were available, but not the variables generated during the follow-up, which would have had to be modeled first in a separate preprocessing step severely limiting reproducibility (e.g., individual drug exposures retrospectively derived by population pharmacokinetic modeling or parameters from tumor growth models relying on measurements after the study baseline). However, it is this particular situation that further emphasizes the importance of interpretable machine learning (IML): now we are all the more interested in which variables (with different availability) contribute to what extent and in which way to the prediction. If we want to describe ML procedures in an interpretable fashion, this is generally possible from two perspectives. First, the procedure can be monitored during the modeling process itself. For example, this could mean to describe how hyperparameters are determined. In their supplementary materials, Chan et al. show coefficient solution paths together with the optimal estimated coefficients selected by the tuning parameter λ in least absolute shrinkage and selection operator (Lasso) regression or by the optimal boosting step number in boosting regression.2 It is also conceivable from this perspective to visualize, for example, how the number of split candidates of a random forest is determined. However, these metrics can only hardly be interpreted in a clinical sense; nevertheless, they can of course be reassuring if they match with the original analysis when reproducing this analysis. The second perspective, on the other hand, looks at the influence of predictor variables on the prediction itself. Whereas new methods for interpretability are published at breakneck speed,3 we restrict ourselves here first for illustration to two catchy and intuitive methods, the variable importance (synonym: feature importance) and the partial dependence plots (synonym: marginal means, predictive margins, and marginal effects). Variable importance (VI) indicates the relevance of a single predictor variable for the overall accuracy of the prediction (or vice versa the prediction error). This importance metric can be determined independently of the type of the (ML) model and thus also allows for comparing different methods directly. The idea behind this is that the prediction error increases after permutation of the values in the predictor variable.5 It is straightforward that the prediction error will increase more for an important variable than for less important ones if the relationship between predictor and outcome is broken down. Figure 1a shows which variables in which order contribute how much to the prediction for the Lasso regression, boosting, and random survival forests examined as examples adapted from the original manuscript.2 In principle, a cutoff in variable importance could also be used for preselecting predictors for another subsequent model, however, this is not done here. For our comparison to the original manuscript, it is also interesting to see which variables in our set of predictor variables are now predictively important. For example, the top six predictor variables of the random survival forests are clinically well-known and expected variables to influence prognosis in lung cancer. This VI plot illustrates the extent, but not the direction, in which the predictor influences the prediction. This marginal effect of a predictor on the outcome can be determined via the partial dependence plot6 from a previously fitted model. Although linear models yield linear relationships, it can be more complex for nonlinear or nonparametric methods, such as random forests. For the top eight variables from the random survival forest, Figure 1b shows how the outcome prediction is influenced by alternating the influence variable on the x-axis. In addition to the functional relationship (curve shape), this also shows the strength in predicting the outcome. Consistent trajectories are desirable when pursuing reproducible results, which must also appear plausible from a clinical point of view. In a clinically plausible manner, higher risks resulted from histologic classification as squamous, higher burden of disease (ECOG status [Eastern Cooperative Oncology Group]), more metastatic sites, higher levels of lactate dehydrogenase (LDH), or lower albumin levels. All these considerations of predictive values can describe ML methods more transparently and be additionally considered during reproduction—but they are not basic requirements (such as, e.g., the use of identical software algorithms and random seeds). Among the many possible modeling methods,8 the so-called split model approach appears intuitively understandable. One develops a prognostic model M in each of the two subgroups of patients under atezolizumab and docetaxel (M1 and M2, respectively). For each patient under any allocation, the CATE can then be determined from the difference in predictions of CATE ^ i = Y i ∗ M 1 - Y i ∗ M 2 , where Y i ∗ represents the individual risk for mortality. Equation 1 thus shows the individual differences in expected risks associated with the two treatments-positive values of CATE illustrate higher risks under atezolizumab (estimated with M1), whereas negative values reflect a higher risk under docetaxel (estimated with M2). A modified waterfall plot9 can clarify the distribution and also make the prediction for an individual patient therein apparent to the clinician (Figure 2a). In addition, HTE modeling is also an ideal showcase to demonstrate how the complex CATE estimates can be made interpretable. For this purpose, a so-called surrogate model can be fitted downstream, which considers the estimated CATEs as the new dependent variable and the set of original predictors of the respective patients as independent variables. Naturally, “fit-the-fit” is a synonymously used term. For this purpose, one can, for example, develop a regression tree and use it to identify important variables (with associated cutoffs) that trigger the complexly determined individual probability of success (CATE) under atezolizumab and docetaxel (Figure 2b). We see that atezolizumab is generally superior on average and for the clear majority of patients. The few explanatory variables identified from a (pruned) regression tree primarily reveal how much a patient in the respective tree branch will benefit from atezolizumab. If these variables are taken to define subgroups, for example, a differential prognosis depending on them may help to classify the outcome clinically. A modified upset plot10 visualizes these relationships in the form of a familiar forest plot (Figure 2b). In patients without high programmed death-ligand 1 (PD-L1)-expression, moderate LDH levels, and less than five metastatic sites, atezolizumab appears to be less beneficial than in the reverse case, where the benefit of atezolizumab appears to be increased for a higher ECOG performance status indicating a larger burden of disease. It should also be noted that all of the presented procedures were used to illustrate the ML models; performance measures for internal and external validity were outside the scope of this commentary. As a conclusion, it is fundamentally important to make ML methods interpretable in order to enhance their comprehensibility and thus acceptance in clinical practice, but also to facilitate reproducibility as a confirmatory step along the way. In our re-analysis of a manuscript based on clinical trial data, we have clarified basic principles and exemplified which methods can be used for each purpose. It would be desirable if such methods were used routinely. This is a fundamental step toward the implementation of such procedures, for example, as decision support in clinical practice with all the legal and ethical implications. With the constantly growing number of possible methods of IML, we like to highlight that further developments can be particularly useful if they are developed together with clinicians, are intuitively understandable, and can be applied independently of the model’s nature. This publication is based on research using data from data contributor Hoffmann-La Roche that has been made available through Vivli, Inc. Vivli has not contributed to or approved, and is not in any way responsible for, the contents of this publication. The authors declared no competing interests for this work.