Causation Entropy Method for Covariate Selection in Dynamic Models
Jared Elinger, Jonathan Rogers · 2021
When constructing models from data, it is often desirable to employ regression techniques that identify the important predictors of model behavior. This process of covariate selection enables construction of models with reduced size that may also be more accurate. LASSO and elastic net regularization are common techniques that are employed to shrink the size of a model by eliminating covariates that do not have a significant effect on predictive performance. However, these methods are subject to well-known limitations in that shrinkage performance must be controlled through the tuning of hyperparameters. This paper introduces a new technique for covariate selection for discrete-time dynamic models using an information theoretic quantity called causation entropy. The algorithm selects important state transition functions in the dynamic model from a set of candidates through the specially-formulated Causation Entropy Matrix (CEM). Unlike other covariate selection methods, the CEM technique does not require tuning but is seen to produce analogous results to LASSO and elastic net in many cases. While the basic structure of the CEM was introduced in prior work, this paper evaluates model shrinkage performance in the presence of varying levels of noise and training data. Performance comparisons are shown between the CEM method and LASSO and elastic net, highlighting the tradeoffs in terms of the effect of data length and robustness to noise. Overall, the CEM method is shown to be a useful technique for covariate selection in cases where limited training data is available and/or noise in the data is relatively low.