Bayesian Generalized Kernel Mixed Models

Zhihua Zhang, Guang Dai and Michael I. Jordan

JMLR 2011

Summary of contributions

• Propose generalized kernel models (GKMs) as a framework in which sparsity can be given an explicit treatment and in which a fully Bayesian methodology can be carried out

• Data augmentation methodology to develop a MCMC algorithm for inference

• Approach shown to be related Gaussian processes and provide a flexible approximation method for GPs

Bayesian approach for kernel supervised learning

• The form of the regressor or classifier is given by

• For a Mercer kernel, there exists a corresponding mapping (say ), from the input space , such that

• This provides an equivalent representation in the feature space, where,

Generalized Kernel Models

Prior for regression coefficients

Sparse models

• Recall that the number of active vectors is the number of non-zero components of– We are thus interested in a prior for which

allows some components of to be zero

Methodology

For the indicator vector

Graphical model

Inference

• Gibbs for most parameters• MH for kernel parameters• Reversible jump Markov Chain for – takes 2^n distinct values– For small n, posterior may be obtained by calculating

the normalizing constant by summing over all possible values of

– For large n, a reversible jump MC sampler may be employed to identify high posterior probability models

Automatic choice of active vectors

• We generate a proposal from the current value of by one of the three possible moves:

Prediction :

Sparse Gaussian process for classification

Given a function , then is a Gaussian process with zero mean and covariance function and vice versa.

Also,

Sparse GP classification

Results