Gibbs Sampling Methods for Stick-Breaking priors Hemant Ishwaran and Lancelot F. James 2001...
Transcript of Gibbs Sampling Methods for Stick-Breaking priors Hemant Ishwaran and Lancelot F. James 2001...
Gibbs Sampling Methods for Stick-Breaking priors
Hemant Ishwaran and Lancelot F. James2001
Presented by Yuting Qi
ECE Dept., Duke Univ.
03/03/06
Overview
Introduction What’s Stick-breaking priors?
Relationship between different priors Two Gibbs samplers
Polya Urn Gibbs sampler Blocked Gibbs sampler
Results Conclusions
Introduction
What’s Stick-Breaking Priors? Discrete random probability measures
pk: random weights, independent of Zk,
Zk are iid random elements with a distribution H, where H is nonatomic.
.
Random weights are constructed through stick-breaking procedure.
Introduction (cont’d)
Steak-breaking construction:
, i.i.d. random variables.
N is finite: set VN=1 to guarantee . pk have the generalized Dirichlet distribution which
is conjugate to multinomial distribution. N is infinite:
Infinite dimensional priors include the DP, two-parameter Poisson-Dirichlet process (Pitman-Yor process), and beta two-parameter process.
0 1v1
1-v1
(1-v1)(1-v2)
v2(1-v1) v3(1-v1) (1-v2)…
),Beta(~ kkk baV
Pitman-Yor Process,
Two-parameter Poisson-Dirichlet Process: Discrete random probability measures
Qn have a GEM distribution
Prediction rule (Generalized Polya Urn characterization):
A special case of Stick-breaking random measure:
1
),(n
Zn nQPy
abanabaWn ),1,0[),,1Beta(~
kabbaa kk ,1
Generalized Dirichlet Random Weights
Finite stick-breaking priors & GD: Random weights p=[p1,..,pN] constructed from
a finite Stick-breaking procedure
is a Generalized Dirichlet distribution (GD). The density for p is
1
1
12111
1
,1,...2,),Beta(~,)1()1)(1(,N
kkN
kkkkkk
pp
NkbaVVVVVpVp
f(p1,..,pN)=f(pN | pN-1,…, p1) f(pN-1 | pN-2,…, p1)…f(p1)
ak=k, bk=k+1+…+N
Generalized Dirichlet Random Weights
Finite dimensional Dirichlet priors: A random measure
with weights, p=(p1,…,pN)~Dirichlet(1,…, N),
p has a GD distribution w/ ak=k, bk=k+1+…+N.
Connection: all random measures based on Dirichlet random weights are Stick-breaking random measure w/ finite N.
Truncations
Finite Stick-breaking random measure can be a truncation of . Discard the N+1, N+2,… terms in , and
replace pN with 1-p1-…-pN-1. It’s an approximation. When as a prior is applied in Bayeisan
hierarchical model,
the Bayesian marginal density under the truncation is
Truncations (cont’d)
If n=1000, N=20, =1, then ~10^(-5)
Polya Urn Gibbs Sampler
Stick-breaking measures used as priors in Bayesian semiparametric models,
Integrating over P, we have
Polya Urn Gibbs sampler: (a)
(b)
Blocked Gibbs Sampler
Assume the prior is a finite dimensional , the model is rewritten as
Direct Posterior InferenceIteratively draw values
Values from joint
distribution of
Each draw defines a random
measure
Blocked Gibbs Algorithm
Algorithm: Let denote the set of current m unique values of
K,
Comparisons
In Polya Urn Process, in one Gibbs iteration, each data inquires existing m clusters & a new cluster one by one. The extreme case is each data belongs to one cluster, ie, # of cluster equals to # of data points.
In Blocked Gibbs sampler, in one Gibbs iteration, all n data points inquire existing m clusters & N-m new different clusters. That’s the infinite un-present clusters in Polya Urn process is represented by N-m clusters in Blocked Gibbs sampler. Since # of data points is finite, once N>=n, N possible clusters are enough for all data even in the extreme case where each data belongs to one cluster.
In this sense, Blocked Gibbs sampler is equivalent to Polya Urn Gibbs sampler.
Results
Simulated 50 observations from a standard normal distribution.