Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian...
Transcript of Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian...
![Page 1: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/1.jpg)
Simulation from Distributions
![Page 2: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/2.jpg)
Sampling Distributions can be simulated from (!) Different distributions require special treatment There are general rules of course (rejection,
composition, inversion) I will cover the basic methods here for common
distributions
![Page 3: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/3.jpg)
Random Number generation All distribution simulation is based on random
number generation Fundamental is the generation of U(0,1) variates: a
random number is a simulation from this distribution.
However all random number generators are not really random as they are constructed mathematically from fixed sequences of numbers that essentisally mimic randomness
![Page 4: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/4.jpg)
Linear congruential generator (LCG) Recursive sequence:
m modulus c increment is the seed All non-negative integersTo get the new value simply compute
1( )(mod )i iZ aZ c m−= +
0Z
1from ( ) /i iZ remainder aZ c m−= +
![Page 5: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/5.jpg)
Random (uniform) Numbers To get the random number set
and thenAs you can see the sequence is fixed but if the a, c, m are chosen well the resulting random numbers will look like a sample from a uniform. • Scaling: for a uniform on different range just multiply
by the range: e.g. for just multiply• For discrete uniform compute set
/i iU Z m=(0,1)iU U
* (0,10)iU U 10* iU( , )ix DU a b
1b a− +
( 1)
. is an interger parti ix a b a U
where
= + − +
![Page 6: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/6.jpg)
Distributions Sampling from any distribution is based on random
numbers Some basic methods:
Inverse transform Composition Convolution Rejection
![Page 7: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/7.jpg)
Inverse Transform Target density: CDF: Denote as the inverse of and
Algorithm:1. Generate2. Set
( )Xf x
( ) Pr(X u)F u = ≤1F − F
1Pr( ) Pr( ( ) ) Pr( ( )) ( )as (0,1)
X x F U x U F x F xU U
−≤ = ≤ = ≤ =
(0,1)iU U1( )i iX F U−=
![Page 8: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/8.jpg)
Exponential example Density and CDF:
Set and solve Then
Algorithm: 1) 2) set
1( ) exp( / )
( ) 1 exp( / )
f x x
F x x
ββ
β
= −
= − −
( )u F x=
1( ) ln(1 )and lnF u u
X uββ
− = − −= −
(0,1)iU U< − lni iX Uβ= −
![Page 9: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/9.jpg)
Some basic intuitive examples Bernoulli
Weibull
Discrete uniform
(0,1)If , set 1, otherwise 0
i
i
U UU p X X< −
≤ = =
1 1/
1/
( ) [ ln(1 )] [ ln( )]i i
F u uSet X U
α
α
ββ
− = − −= −
( 1)
. is an interger parti ix a b a U
where
= + − +
![Page 10: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/10.jpg)
Composition Ideal for distribution mixtures
where
Algorithm:
1) Generate a positive random integer J such that
2) Given , generate X with distribution
Examples include the Laplace (double exponential)
1( ) ( )X j j
jf x p f x
=
=
10; 1j j
jp p
∞
=
≥ =
Pr( ) for 1,2,3.....jJ j p j= = =
J j= jF
![Page 11: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/11.jpg)
Convolution If an random variable can be expressed as a sum of IID
variables then it can be generated from a sum Algorithm 1. Generate2. Set Examples: binomial distribution (convolution of IID Bernoullis) Negative binomial (convolution of IID geometrics)Chi-squared K (convolution of IID Chi-squared df=1)Ga(a,b) b*convolution of a IID exponential(1)s (a integer)
1 2 3 4 5 6, , , , , ,........., mY Y Y Y Y Y Y
1 2 3 4 5 6 .........i mX Y Y Y Y Y Y Y= + + + + + + +
![Page 12: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/12.jpg)
Rejection sampling Basically we specify a function that majorizes the
density ie
Define
Must be able to easily generate a variate from r(x)
( ) ( ) t x f x x≥ ∀
( ) ( ) 1
( ) ( ) / is a density.
c t x dx f x dx
but r x t x c
∞ ∞
−∞ −∞= ≥ =
=
![Page 13: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/13.jpg)
Rejection algorithm Generate Y from density r Generate If then set otherwise return to
step 1 Algorithm loops until we get a valid X
(0,1)iU U< −( ) / ( )iU f Y t Y≤ X Y=
![Page 14: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/14.jpg)
Beta example Rejection:Beta(4,3) with density
Mode of density is 0.6 and
So is just the uniform density
Algorithm:
Can be generated from gamma distributions also
3 2( ) 60 (1 ) 0 x 1f x x x if= − ≤ ≤
(0.6) 2.0736 ( ) 2.076 if 0 1
( ) majorises ( )
fhence set t x xthen t x f x
== ≤ ≤
( )r x
Y (0,1)If (Y) / 2.076 set X = Y otherwise reject YU and U
U f< −
≤
![Page 15: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/15.jpg)
Figure
![Page 16: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/16.jpg)
Special distributions GaussianBox-Mueller or Polar Marsaglia Log normalTransform of normal distribution GammaComplex rejection sampling Students tFrom Gaussian and chi-squared distributions FFrom chi-squared distributions
![Page 17: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/17.jpg)
Correlated random variables Can exploit conditioning i.e. Generate
Not very useful
11
2 2 1
3 3 1 2
1 2 1
marginal distribution (. | )(. | , )
(. | , ,......., )
X
n n n
X F
X F XX F X X
X F X X X −
< −
< −< −
< −
![Page 18: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/18.jpg)
Multivariate Normal generation The MVN is relatively easy to simulate from given the
covariance matrix is known Define
Algorithm:1) Generate IID N(0,1) variates2) For i=1, 2, …..,n let
1 2 3 4(X ,X ,X ,X ,....,X ) ( , )and the covariance matrix can be decomposedby Cholesky: =CC where is the ( , ) th element of C.
Tn n
Tij
N
c i j
= Σ
Σ
X μ
1 2 3 4, , , , ...., nZ Z Z Z Z
1
i
i i ij jj
X c Zμ=
= +
![Page 19: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/19.jpg)
Adaptive Rejection (ARS) Rejection sampling is simple but can be inefficient The Beta example used a uniform constant: this yields
a lot of rejections. It would be better to get functions (t(x)) that wrap
closer to the target (f(x)) But they must be easy to simulate from
![Page 20: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/20.jpg)
Example
![Page 21: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/21.jpg)
Adaptive Rejection Sampling The basic idea is that getting a good fit to an envelope
could be adaptive: i.e. as you sample the sampler adjusts to the form of the function
It is designed for a log concave function: most likelihoods are log concave (exponential family are)
ARS proceeds by majorising the log densitywith piecewise linear segments instead of a uniform density. The segments are adjusted as the sampler proceeds. This can be done by using log exponential distributed linear segments An under function can also be used to help acceptances
( ) log( ( ))g x f x=
![Page 22: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/22.jpg)
ARS
![Page 23: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/23.jpg)
ARMS If the density is not log concave it is still possible to
use ARS but an extension for non –concavity includes a metropolis step: adaptive rejection Metropolis Sampling (ARMS) is an automatic method that can tune the sampler to any concave or non-concave function.
On software platforms such as BUGS: ARS and ARMS are implemented and represent the ‘adaptive phase’ .
![Page 24: Simulation from Distributions - MUSCpeople.musc.edu › ~abl6 › BMTRY789-19 Advanced Bayesian Comput… · Random Number generation ... Must be able to easily generate a variate](https://reader034.fdocuments.us/reader034/viewer/2022042404/5f19168381e11359c7481e77/html5/thumbnails/24.jpg)
Slice Sampling Radford Neal introduced this idea:
Neal, Radford M. (2003). "Slice Sampling". Annals of Statistics. 31 (3): 705–76.
Basically, slices are cut across distributions and sampling is based on the slice width
It can be more efficient than MH and can be incorporated within an McMC algorithm