Monte Carlo Methods and Statistical Computing: My …home.iitk.ac.in/~kundu/tirupati-2014-1.pdfMonte...
Transcript of Monte Carlo Methods and Statistical Computing: My …home.iitk.ac.in/~kundu/tirupati-2014-1.pdfMonte...
Monte Carlo Methods and Statistical Computing:My Personal Experience
Debasis Kundu
Department of Mathematics & Statistics
Indian Institute of Technology Kanpur
November 29, 2014
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Limitations:
1. I must admit that the topics I am going tocover are definitely not exhaustive.
2. Topics are purely of my own interest whichhave developed over the last 30 years.
3. I am not going to describe any statisticalpackage.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Advantages:
1. Packages have their own problems.
2. Different packages can give different answerseven on a relatively simple problem.
3. We should know the limitations of thepackages.
4. I will try to provide a general approach
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Journals:
1. Journal of Computational and GraphicalStatistics.
2. Computational Statistics and Data Analysis.
3. Journal of Statistical Computation andSimulation.
4. Statistical Computing
5. Communications in Statistics - Simulationand Computation
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Books:
1. Simulation by Sheldon Ross
2. Nonuniform Random Deviate Generator, L.Devroye
3. Simulation modelling and analysis, Law andKelton Journal of Statistical Computation andSimulation.
4. Statistical Computing: J.F. Keneddy and R.Gentle
5. Statistical Computing, D. Kundu and A. Basu
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Monte Carlo Method: Definition
1. A broad class of numerical alogrithm dependson repeated random sampling.
2. If it is not possible to obtain the exactanalytical solution often Monta Carlo methodcan be used to provide a very good approximatesolution
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Monte Carlo Method: A brief history
1. It was invented by Stanislaw Ulam, a famousPolish Mathematician, in the late 1940.
2. John von Neumann first wrote the computercode to perform Monte Carlo simulations
3. Metropolis gave this name
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Where it can be used?
1. Calculating the area below a curve.
2. Calculating multidimensional integration.
3. Optimization.
4. Analyzing any complicated stochastic system(model).
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Examples
Suppose we want to compute
∫ b
a
e−x2dx .
Or suppose we want to compute
∫ b1
a1
. . .
∫ bk
ak
f (x1, . . . , xk)dx1 . . . dxk .
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Examples:Contd.
Suppose we want to find the maximum or minimumof the following function
f (x1, . . . , xk),
where a1 ≤ x1 ≤ b1, . . . , ak ≤ xk ≤ bk .
Or suppose we want to analyze the followingnon-linear model
y(x1, . . . , xk) = f (x1, . . . , xk , θ) + e.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Back Ground
1. Knowledge of Basic Probability.
2. Discrete and Continuous random variables.
3. Stochastic models.
4. Generation of random numbers.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Knowlwedge of Basic Probability
1. Idea of a random experiment.
2. Basic idea of convergence of randomvariables.
3. Weak and strong law of large numbers.
4. Central limit theorem.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Discrete Random Variables
1. Uniform.
2. Binomial.
3. Geometric.
4. Poisson.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Continuous Random Variables
1. Uniform.
2. Exponential.
3. Normal.
4. Gamma.
5. Log-concave probability density function
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Generation of Random Numbers
First we need to know how to generate Uniformrandom numbers. This is the most basic problem.In this respect we use group theory results andmachine powers.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Generation of Non-Uniform Random Numbers
The most popular method is the inversetransformation. The following result can be used. IfX is a random variable with the distributionfunction F (x), then F (X ) follows uniformdistribution. Therefore
X = F−1(U)
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Generation of Discrete Random Numbers
All the discrete distributions can be generated usinginverse transformation method. SupposeP(X = ai) = pi , for i = 1, 2, . . .. Without loss ofgenerality we can assume a1 < a2 < . . .. Draw auniform random number say u, if∑k−1
i=1 pi < u <∑k
i=1 pi , then X takes the value ak .
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Generation of Continuous Random Numbers
Many continuous random variables can begenerated using inverse transformation method, forexample exponential, Weibull, generalizedexponential distributions etc. On the other handseveral well known distribution cannot be obtainedusing inverse transformation method. For examplenormal, gamma etc.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Generation of Continuous Random Numbers
If a continuous distribution cannot be generatedusing inverse transformation method, one of themost useful method is the acceptance rejectionmethod. The idea is as follows. If we want togenerate from f (x), try to find g(x), from whichgeneration is simple so that it satisfies the following
f (x) ≤ cg(x).
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Acceptance Rejection Method: Algorithm
1. Generate Y from g(x).
2. Generate a uniform random vaiable U .
3. If U ≤ f (Y )/cg(Y ), set X = Y , otherwisereturn to 1.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Acceptance Rejection Method: Theorem
Theorem:
1. The random variable generated by thismethod has density function f (x)
2. The number of iterations of the algorithmthat are needed is a geometric random variablewith mean c ,
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Acceptance Rejection Method: Example
Example 1:
Suppose we want to generate from
f (x) = 20x(1− x)3; 0 < x < 1.
Takeg(x) = 1, 0 < x < 1.
c = 135/64.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Acceptance Rejection Method: Example
Example 2:
Suppose we want to generate from
f (x) =2√πx1/2e−x ; 0 < x < ∞.
Take
g(x) =2
3e−2x/3 0 < x < ∞.
and
c =33/2
(2πe)1/2.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Acceptance Rejection Method: Example
Example 3:
Suppose we want to generate from
f (x) =2√2π
e−x2/2; 0 < x < ∞.
Takeg(x) = e−x ; 0 < x < ∞.
andc =
√2e/π.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Very Simple Example
Consider the following simple linear regression model
Y = Xb+ e
We know the LSE’s can be obtained as
b̂ = (XTX)−1
XTY.
We have a complete very nice theory when all thecomponents of the errors are i.i.d. normal randomvariables.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Very Simple Example: Contd.
Consider some slightly different conditions of thesame model.
1. What will happen if the errors are notnormal?
2. What will happen if the errors are heavy tail?
3. What will happen if there are outliers?
4. What will happen if the errors are correlated?
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Very Simple Example: Contd.
In all these cases Monte Carlo Method can be usedto asses the performances of the estmimators. It isvery simple also.
1. Generate e
2. Generate Y.
3. Calculate b̂.
4. Repeat step 1 to step 4, several times.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example
Consider the following simple linear regression model
Y = Xb+ e
Suppose we want to estimate b by minimizing theleast absolute errors i.e.
b̂ = argmin|Y − Xb|.
Theories are quite complicated. All the results areasymptotic in nature.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example
Consider the following non-linear regression model
Y = f (X, θ) + e
Here f is a known function the vector X is alsoknown, the paramete vector θ is unknown. Theproblem is to estimate the parameter vector θ,based on a sample of size n.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example
Natural estimators will be
θ̂ = argmin|n∑
i=1
Yi − f (Xi , θ)|2.
or
θ̂ = argmin|n∑
i=1
Yi − f (Xi , θ)|.
Theories are quite complicated. All the results areasymptotic in nature.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example
Monte Carlo method can be used to asses theperofrmance of the estimators.
Based on the Monte Carlo method the biasesand the mean squared errors can be calculated.
Based on Bootstrap method confidence intervalsalso can be obtained.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example: Importance Sampling
In Bayesian analysis often we need to compute theposterior mean as follows:
θ = E (h(X)) =
∫h(x)f (x)dx.
Here f (x) is the PDF of X, and x can be a veryhigh dimensional. In Bayesian analysis f (x) is theposterior density function.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example: Importance Sampling
Monte Carlo simulation technique can be used toapproximate the value of θ as follows:
θ̂ =1
N
N∑
i=1
h(Xi),
here X1, . . . ,XN is a random sample of size N fromf (X).
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Example: Importance Sampling
Often it is observed that it is not very easy togenerate samples from f (x).
θ =
∫h(x)f (x)dx =
∫h(x)f (x)
g(x)g(x)d(x).
θ̂ =1
N
N∑
i=1
h(xi)f (xi)
g(xi),
here X1, . . . ,XN is a random sample of size N fromg(X).
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Important Issues
1. Finding Maximum likelihood estimators in ageneral problem.
2. Finding least squares estimators of linearregresssion model when the design matrix isclose to a singular matrix
3. Non-linear regression model if the number ofparameters are very high
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
MLE
It basically invloves maximizing a function of theform:
f (θ1, . . . , θp)
Standard method is to use Newton-Raphsonmethod:
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Newton-Raphson Method
Assuming sufficiently smooth f (θ), we want to solve
∂f (θ)
∂θ= 0
Standard method is to use Newton-Raphsonmethod. Using Taylor series expansion, it can beeasily obtained:
θ(k+1) = θ(k) −[∂2f (θ(k))
∂θ∂θT
]−1∂f (θ(k))
∂θ
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Profile Likelihood Method
1. For fixed θ1, . . . , θk , try to maximize withrespect to θk+1, . . . , θp
2. Maximize with respect to θ1, . . . , θk .
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
EM Algorithm
Suppose the data are coming from a mixture model,and we compute the MLEs of the unknownparameters
f (x) =k∑
j=1
πj fj(x ; θj),
πj ≥ 0,∑k
j=1 πj = 1.
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Mixture Model: MLE
Based on a random sample x1, . . . , xn, we want tocompute the MLEs of the unknown parameters
L(π, θ) =n∏
i=1
k∑
j=1
πj fj(xi ; θj).
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Missing Value Problem
We treat this as a missing value problem
1. Assume the data are of the form (x , δ)
2. Compute E (δ|Data)3. Continue the process
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Copula Method
Any multivarite distribution can be written uniquelyas follows:
F (x1, . . . , xp; θ) = C (F1(x1; θ1), . . . , Fp(xp : θp); γ)
First estimate the marginal parameters, and thenestimate the copula parameters
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Non-linear regression
Consider the following model
y(t) =
p∑
k=1
[Ak cos(ωkt) + Bk sin(ωkt)] + e(t)
Estimate the unknown parameters
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Very important areas
Bayesian comutation: mainly MCMC
Classification problem
Small n large p problem
Non-parametric regression
Functional data analysis
Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp