Variable Selection and Decision Trees: The DiVaS and ALoVaS Methods

Abstract

In this thesis we propose a novel modification to Bayesian decision tree methods. We provide a historical survey of the statistics and computer science research in decision trees. Our approach facilitates covariate selection explicitly in the model, something not present in previous research. We define a transformation that allows us to use priors from linear models to facilitate covariate selection in decision trees. Using this transform, we modify many common approaches to variable selection in the linear model and bring these methods to bear on the problem of explicit covariate selection in decision tree models. We also provide theoretical guidelines, including a theorem, which gives necessary and sufficient conditions for consistency of decision trees in infinite dimensional spaces. Our examples and case studies use both simulated and real data cases with moderate to large numbers of covariates. The examples support the claim that our approach is to be preferred in large dimensional datasets. Moreover, our approach shown here has, as a special case, the model known as Bayesian CART.