Sparse Online Variational Bayesian Inference

Abstract: This work aims to study variational Bayesian inference for sparse regression. Sparsity promoting priors have proven to be very successful in regression scenarios since it helps to select meaningful features and avoid overfitting (e.g., LASSO regression or total variation regularization in imaging). We focus on a general class of shrinkage priors that can be represented as a scale mixture of normal distributions with a generalized inverse Gaussian distribution and includes such priors as Laplace, Generalized Jeffrey's, Student-t, and others. However, since shrinkage priors are non-Gaussian, a fully Bayesian solution becomes very expensive, requiring Markov Chain Monte Carlo (MCMC) methods. To alleviate this, we employ a variational approach that leverages a generalization of the expectation-maximization algorithm to recover the best Gaussian approximation to the sparsity-promoting posterior. This approach turns out to be especially fast and scalable in the case of linear models, where it provides approximate uncertainty quantification (UQ) for a substantially smaller cost than fully Bayesian approaches yet keeping comparable accuracy. Besides, the proposed approach supports online inference to process the data in batches and strategies for online hyperparameter estimation. The high performance in terms of the variable selection and UQ is demonstrated for complex real and simulated data examples where it competes against MCMC based methods as well as the other approximate approaches.

Speaker’s Bio: Vitaly is a Statistics PhD student in the Department of Mathematics at the University of Manchester, joint with The Alan Turing Institute. Before coming to Manchester, he obtained BSc and MSc in Applied Mathematics and Physics from the Moscow Institute of Physics and Technology. He also benefited by finishing the Data Science MSc track from the Skolkovo Institute of Science and Technology. He has industrial research experience obtained through taking part in a number of research and development projects in mathematical modelling of wireless telecommunication networks, aspiring to combine and adapt the cutting-edge research breakthroughs with the industry needs.

Currently, he is working on methods for online regression and adaptive experimental design techniques, as well as, more generally, UQ for machine learning algorithms.