Markov Chain Sampling Methods for Dirichlet Process Mixture Models

R.M. Neal

Summarized by Joon Shik Kim12.03.15.(Thu)

Computational Models of Intelligence

Abstract

• This article reviews Markov chain methods for sampling from the posterior distribu-tion of a Dirichlet process mixture model and presents two new classes of methods.

• One new approach is to make Metropolis-Hastings updates of the indicators specify-ing which mixture component is associ-ated with each observation, perhaps sup-plemented with a partial form of Gibbs sampling.

Chinese Restaurant Process (1/2)

Chinese Restaurant Process (2/2)

Introduction (1/2)

• Modeling a distribution as a mixture of simpler distribution is useful both as a nonparametric density estimation method and as a way of iden-tifying latent classes that can explain the depen-dencies observed between variables.

• Use of Dirichlet process mixture models has be-come computationally feasible with the devel-opment of Markov chain methods for sampling from the posterior distribution of the parameters of the component distribution and/or of the asso-ciations of mixture components with observa-tions.

Introduction (2/2)

• In this article, I present two new ap-proaches to Markov chain sampling.

• A very simple method for handling non-conjugate priors is to use Metropolis-Hast-ings updates with the conditional prior as the proposal distribution.

• A variation of this method may some-times sample more efficiently, particularly when combined with a partial form of Gibbs sampling.

Dirichlet Process Mixture Models (1/5)

• The basic model applies to data y1,…,yn which we regard as part of an indefinite exchangeable sequence, or equivalent, as being independently drawn from some unknown distribu-tion.

Dirichlet Process Mixture Models (2/5)

• We model the distribution from which the yi are drawn as a mixture of dis-tributions of the form F(θ), with the mixing distribution over θ given G. We let the prior for this mixing distri-bution be a Dirichlet process, with concentration parameter α and base distribution G0.

Dirichlet Process Mixture Models (3/5)

Dirichlet Process Mixture Models (4/5)

Dirichlet Process Mixture Models (5/5)

• If we let K go to infinity, the condi-tional probabilities reach the follow-ing limits:

Gibbs Sampling when Conjugate Priors are used (3/4)

Nested CRP

Day 1 Day 2 Day 3