Presented by: Flavia Tsang M.A.Sc Student 12 th March 2004

Microsimulation And Modelling Microsimulation And Modelling Applications: Applications:

Methods, Issues and AnalysisMethods, Issues and Analysis(MAMAMIA)(MAMAMIA)

Module 1 – Unit 3:Module 1 – Unit 3: Estimation and Estimation and

ValidationValidationPresented by:Presented by:Flavia TsangFlavia Tsang

M.A.Sc StudentM.A.Sc Student1212thth March 2004 March 2004

OutlineOutline

Part I Part I Simulation-Assisted EstimationSimulation-Assisted Estimation

Part IIPart II Parameter Estimation Strategies for Parameter Estimation Strategies for

Large Scale Urban ModelsLarge Scale Urban Models Part III Part III

Validation of Microsimulation ModelsValidation of Microsimulation Models

Part IPart ISimulation-Assisted Simulation-Assisted

EstimationEstimation

Simulation-Assisted Simulation-Assisted EstimationEstimation

Purpose: Purpose: To examine various methods of To examine various methods of

estimation:estimation: Maximum Simulated LikelihoodMaximum Simulated Likelihood Method of Simulated MomentsMethod of Simulated Moments Method of Simulated ScoreMethod of Simulated Score

To understand the advantages and To understand the advantages and limitations of each form of estimation, limitations of each form of estimation, thereby facilitating our choice among thereby facilitating our choice among methodsmethods

Maximum Simulated Maximum Simulated LikelihoodLikelihood Review - the log-likelihood function is:Review - the log-likelihood function is:

where where

is a vector of parameters, is a vector of parameters,

PPnn(( ) is the (exact) probability of the ) is the (exact) probability of the observed choice of observation nobserved choice of observation n

The summation is over a sample of NThe summation is over a sample of N The maximum likelihood (ML) The maximum likelihood (ML)

estimation is the value of estimation is the value of that that maximizes LL(maximizes LL())

Maximum Simulated Maximum Simulated LikelihoodLikelihood

Since the gradient of LL(Since the gradient of LL() is zero at the ) is zero at the maximum, the ML estimator can also be maximum, the ML estimator can also be defined as the value of defined as the value of at which at which

wherewhere

ssnn(( ) = ) = ln Pln Pnn(()/ )/ is the score for is the score for observation nobservation n

Maximum simulated likelihood (MSL) Maximum simulated likelihood (MSL) has the same formulation as maximum has the same formulation as maximum likelihood (ML) except that simulated likelihood (ML) except that simulated probabilities are used in lieu of the exact probabilities are used in lieu of the exact probabilityprobability


The main issue with MSL arises The main issue with MSL arises because of the log transformationbecause of the log transformation Suppose PhatSuppose Phat is an unbiased simulator is an unbiased simulator

for Pfor Pnn(()) Since log operation is a Since log operation is a nonlinear nonlinear

transformationtransformation, ln Phat, ln Phat is not is not unbiased for ln Punbiased for ln Pnn(())

The bias in the simulator of ln PhatThe bias in the simulator of ln Phat translate into bias in the MSL estimatortranslate into bias in the MSL estimator


To determine the asymptotic properties of To determine the asymptotic properties of the MSL estimator:the MSL estimator: How the simulation bias behaves when the How the simulation bias behaves when the

sample size increases?sample size increases?

depends on the relationship between depends on the relationship between number of draws (R) and sample size (N)number of draws (R) and sample size (N)

If R is fixed, then the MSL estimator does not If R is fixed, then the MSL estimator does not converge to the true parameters, because of converge to the true parameters, because of the simulation bias in ln the simulation bias in ln

If R rises with N, the simulation bas If R rises with N, the simulation bas disappears as N rises without bounddisappears as N rises without bound


In summary, In summary, If R is fixed, then MSL is inconsistentIf R is fixed, then MSL is inconsistent If R rises at any rate with N, then MSL If R rises at any rate with N, then MSL

is consistentis consistent If R rises faster than the root of N, then If R rises faster than the root of N, then

MSL is not only consistent but also MSL is not only consistent but also efficientefficient

Method of Simulated Method of Simulated MomentsMoments

Method of Moments (MOM) is defined as the Method of Moments (MOM) is defined as the parameters that solve the equation:parameters that solve the equation:

wherewhere

ddnjnj is the dependent variable that identified the is the dependent variable that identified the chosen alternative: dchosen alternative: dnj nj = 1 if n chose j, 0 = 1 if n chose j, 0 otherwiseotherwise

zznjnj is a vector of exogenous variable called is a vector of exogenous variable called weightsweights

Methods of Simulated Methods of Simulated MomentsMoments

DDnjnj – P – Pnjnj(( ) are the residuals) are the residuals MOM estimator is the parameter value at MOM estimator is the parameter value at

which the residuals are uncorrelated with which the residuals are uncorrelated with the instruments in the samplethe instruments in the sample

When some instruments are chosen so that When some instruments are chosen so that MOM becomes efficient, such MOM becomes efficient, such instructments are called the ideal instructments are called the ideal instrumentsinstruments

For standard logit model, the explanatory For standard logit model, the explanatory variables can be used as ideal instrumentsvariables can be used as ideal instruments


The method of simulated moment The method of simulated moment (MSM) is obtained by replacing the (MSM) is obtained by replacing the exact probability Pexact probability Pnjnj(()with simulated )with simulated probabilityprobability

The MSM estimator is the value of The MSM estimator is the value of that solvesthat solves


Pnjth enters the equation linearly, Pnjth enters the equation linearly, therefore, Pnjth is unbiased for Ptherefore, Pnjth is unbiased for Pnjnj(())

Since there is no simulation bias in the Since there is no simulation bias in the estimation condition, MSM estimator estimation condition, MSM estimator is consistent even when R is fixedis consistent even when R is fixed

MSM still contains simulation noise MSM still contains simulation noise (variance due to simulation). This (variance due to simulation). This noise becomes smaller as R risesnoise becomes smaller as R rises

Disadvantage of MSM is that it loss Disadvantage of MSM is that it loss efficiency when non-ideal weights are efficiency when non-ideal weights are usedused


In summary,In summary, Disadvantage of MSM is that it loss Disadvantage of MSM is that it loss

efficiency when non-ideal weights are efficiency when non-ideal weights are usedused In case the instrument are simulated with In case the instrument are simulated with

bias and the instruments are not ideal, the bias and the instruments are not ideal, the MSM estimator has same properties as MSL MSM estimator has same properties as MSL except that it is not asymptotically efficientexcept that it is not asymptotically efficient

Yet, it overcome the problem of MSLYet, it overcome the problem of MSL MSM is consistent when R is fixedMSM is consistent when R is fixed

Method of SimulatedMethod of Simulated ScoresScores

The method of simulated score The method of simulated score (MSS) estimator is the value of (MSS) estimator is the value of that solvesthat solves

where snth is a simulator of the where snth is a simulator of the scorescore

The score can be rewritten as:The score can be rewritten as:

Method of Simulated Method of Simulated ScoresScores

An unbiased simulator for the second An unbiased simulator for the second term term PPnjnj// is easily obtained by taking is easily obtained by taking the derivative of the simulated the derivative of the simulated probabilityprobability

The difficulty arises in finding an The difficulty arises in finding an unbiased simulator for the first term unbiased simulator for the first term 1/P1/Pnjnj(()) Simply taking the inverse of the simulated Simply taking the inverse of the simulated

probability does not provide an unbiased probability does not provide an unbiased estimatorestimator


Need to obtain 1/PNeed to obtain 1/Pnjnj(():): Consider drawing balls from an urn that Consider drawing balls from an urn that

contains many balls of different colours. contains many balls of different colours. Suppose the probability of obtaining a Suppose the probability of obtaining a red ball is 0.2. How many draws would red ball is 0.2. How many draws would it take on average to obtain a red ball?it take on average to obtain a red ball?

Answer: 1/0.2 =5Answer: 1/0.2 =5 The same idea can be applied to choice The same idea can be applied to choice

probabilityprobability


To obtain 1/PTo obtain 1/Pnjnj(():):

1. Take a draw of the random terms from 1. Take a draw of the random terms from their densitytheir density

2. Calculate the utility of each alternative 2. Calculate the utility of each alternative with this drawwith this draw

3. Determine whether alternative j has the 3. Determine whether alternative j has the highest utilityhighest utility

4. If so, call the draw an accept. If not, call 4. If so, call the draw an accept. If not, call the draw a reject and repeat steps 1 to 3 the draw a reject and repeat steps 1 to 3 with a new draw with a new draw


To obtain 1/PTo obtain 1/Pnjnj(() (cont’d):) (cont’d):

5. Define B5. Define Brr as the number of draws that are as the number of draws that are taken until the first accept is obtained. taken until the first accept is obtained.

Perform step 1 to 4 R times, obtaining BPerform step 1 to 4 R times, obtaining Brr for r =1, … Rfor r =1, … R

The simulator of 1/PThe simulator of 1/Pnjnj(() is:) is:

The simulator is unbiased for 1/PThe simulator is unbiased for 1/Pnjnj(() )


In summary,In summary, no guarantee that an accept will be no guarantee that an accept will be

obtained in a given number of drawsobtained in a given number of draws not continuous in parametersnot continuous in parameters MSS overcomes the problems of MSLMSS overcomes the problems of MSL

For fixed, for fixed R, MSS is consistent and For fixed, for fixed R, MSS is consistent and asymptotically normalasymptotically normal

Part II Part II Parameter Estimation Parameter Estimation

Strategies for Large Scale Strategies for Large Scale Urban ModelsUrban Models

Parameter Estimation Parameter Estimation StrategiesStrategies

Purpose:Purpose: Examine different strategies for Examine different strategies for

parameter estimation in large scale parameter estimation in large scale models; five different strategies will models; five different strategies will be presented:be presented: Limited viewLimited view Piece wisePiece wise SimultaneousSimultaneous SequentialSequential Bayesian SequentialBayesian Sequential

A Modular Modelling A Modular Modelling SystemSystem

There are connections between submodels, There are connections between submodels, representing data flows within the modelling systemrepresenting data flows within the modelling system

The submodels have a certain degree of independence The submodels have a certain degree of independence from each other, and the degree to which they are from each other, and the degree to which they are treated as independent leads to different strategiestreated as independent leads to different strategies

Limited View ApproachLimited View Approach

Focus on the entire modelling Focus on the entire modelling system, ignoring individual system, ignoring individual submodels submodels

The modelling system is run with the The modelling system is run with the input set to observed values and the input set to observed values and the parameters are adjusted until the parameters are adjusted until the modeling system’s outputs closely modeling system’s outputs closely match corresponding observed match corresponding observed valuesvalues

Limited View ApproachLimited View Approach

Advantages: Advantages: Allow the model-builder to concentrate on Allow the model-builder to concentrate on

how the model will be used in application, how the model will be used in application, instead of how the parameters will be instead of how the parameters will be estimatedestimated

Focusing on the entire modelling system is Focusing on the entire modelling system is likely to reveal structure problemslikely to reveal structure problems

Disadvantages:Disadvantages: Cannot make use of “extra data”, which are Cannot make use of “extra data”, which are

synthesized by other submodels (eg. In a synthesized by other submodels (eg. In a nested logit formulation, result of the lower nested logit formulation, result of the lower model informs the upper model)model informs the upper model)

Piece Wise Estimation Piece Wise Estimation

Connections between submodels are Connections between submodels are ignoredignored

The parameter of each submodel are The parameter of each submodel are estimated based on the data that estimated based on the data that directly affect that submodeldirectly affect that submodel

Piece Wise EstimationPiece Wise EstimationAdvantages:Advantages: Breaks the problem into more Breaks the problem into more

manageable piecesmanageable pieces Easer to use extra data during the Easer to use extra data during the

consideration of each submodel consideration of each submodel ( i.e. entirely different data could be used to ( i.e. entirely different data could be used to

inform parameter values, eg. targeted inform parameter values, eg. targeted sample data, stated preference data, and sample data, stated preference data, and even data from a different city)even data from a different city)

Often correspond to well-establish Often correspond to well-establish theories and are often operationalized theories and are often operationalized using fairly simple equationsusing fairly simple equations

Piece Wise EstimationPiece Wise Estimation

Disadvantages:Disadvantages: Sometimes it is impossible to consider a Sometimes it is impossible to consider a

submodel on its own, if no observed data submodel on its own, if no observed data is available to replace the synthesized is available to replace the synthesized datadata

If dependent submodels are non-linear, If dependent submodels are non-linear, this could lead to a bias in outputsthis could lead to a bias in outputs

Combining accurate submodels do not Combining accurate submodels do not guarantee an accurate overall modelling guarantee an accurate overall modelling systemsystem

Simultaneous EstimationSimultaneous Estimation

Overall modelling system is run and Overall modelling system is run and its outputs are compared to various its outputs are compared to various targetstargets

Concurrently, each of the individual Concurrently, each of the individual submodels are also run to process submodels are also run to process the extra data availablethe extra data available

Simultaneous EstimationSimultaneous Estimation

Advantages:Advantages: Overcomes many data availability Overcomes many data availability

problems, since missing data can be problems, since missing data can be synthesize from other submodelssynthesize from other submodels

Particularly appropriate when there is a Particularly appropriate when there is a theoretical reason why a parameter in one theoretical reason why a parameter in one submodel should be identical to a submodel should be identical to a parameter in another submodelparameter in another submodel

Disadvantages:Disadvantages: Computationally intensiveComputationally intensive

Sequential EstimationSequential Estimation

Combine piece wise estimation with Combine piece wise estimation with the limited view approachthe limited view approach

The parameters of individual The parameters of individual submodels are estimated, and then submodels are estimated, and then the overall model is consideredthe overall model is considered

Various parameters are identified as Various parameters are identified as being crucial to the higher level being crucial to the higher level behaviour of the model, and these behaviour of the model, and these parameters are estimated by parameters are estimated by examining the entire modelling systemexamining the entire modelling system


Advantages:Advantages: Various extra data can be used when Various extra data can be used when

estimating the lower level models, but estimating the lower level models, but the highest level of estimation can ignore the highest level of estimation can ignore these datathese data

Observed data not required for all data Observed data not required for all data flows because previously calibrated flows because previously calibrated model can provide synthetic data, and model can provide synthetic data, and

The entire modelling system is adjusted The entire modelling system is adjusted in a systematical way to match observed in a systematical way to match observed datadata


Disadvantages:Disadvantages: Less accurate than simultaneous Less accurate than simultaneous

estimationestimation Error estimates on parameter values are Error estimates on parameter values are

biasedbiased For a parameter that is shared between For a parameter that is shared between

submodels, its value will be determined submodels, its value will be determined by the last estimation procedure; the by the last estimation procedure; the information on the parameter from earlier information on the parameter from earlier estimation will be discardedestimation will be discarded

Bayesian Sequential Bayesian Sequential EstimationEstimation

Allows for a Allows for a prior density functionprior density function to to specify what is already know about certain specify what is already know about certain parameter values (eg. range of acceptable parameter values (eg. range of acceptable parameter values)parameter values)

Advantages:Advantages: The parameter estimates and confidence limits The parameter estimates and confidence limits

from the estimation of the parameters within from the estimation of the parameters within individual submodels could be used when individual submodels could be used when estimating at the highest level of the modelling estimating at the highest level of the modelling systemsystem

Less complex than full simultaneous estimationLess complex than full simultaneous estimation

Part IIIPart IIIValidation of Validation of

Microsimulation Microsimulation ModelsModels

Relationship of Validation, Relationship of Validation, Verification and establishing Verification and establishing

credibilitycredibility

System Conceptual ModelSimulation ProgramCorrect ResultsResults Implemented

Source: Law and Kelton (1991) “Simulation Modelling and Analysis”, McGraw-Hill, Inc.

Validation

Verification

Validation

Establish

Credibility

Analysis and data

Programming

Make model runs

Sell Results to Managem

ent

The Role of ValidationThe Role of Validation

Validation is a proactive, diagnostic effort to Validation is a proactive, diagnostic effort to ensure that the model’s results are reasonable ensure that the model’s results are reasonable and credibleand credible

A formal validation exercise produces an A formal validation exercise produces an extensive battery of tests /measures extensive battery of tests /measures /comparisons/comparisons

Validation is qualitatively distinct from just Validation is qualitatively distinct from just making sure the model is doing what one has making sure the model is doing what one has told it to do (Verification)told it to do (Verification)

Quantitative measures are used for validation, Quantitative measures are used for validation, but the ultimate impact is inherently qualitativebut the ultimate impact is inherently qualitative

Special ChallengesSpecial Challenges

Long Term AnalysisLong Term Analysis Projection for these models extend Projection for these models extend

well into the future, often several well into the future, often several decades beyond the present. There decades beyond the present. There are few sources of “future data” are few sources of “future data” against which to assess against which to assess reasonablenessreasonableness


Monte Carlo NatureMonte Carlo Nature Simulations driven by random inputs Simulations driven by random inputs

will produce random outputwill produce random output Decision-makers, however, dislike Decision-makers, however, dislike

such variation. Almost without such variation. Almost without exception, they would prefer to see exception, they would prefer to see point estimatespoint estimates


Which items to validate?Which items to validate? There is a great mass of information There is a great mass of information

being projected, different portion of being projected, different portion of which are relevant for various analysiswhich are relevant for various analysis

This poses the unavoidable question of This poses the unavoidable question of which particular items to validate, which particular items to validate, given resource constraints, and of how given resource constraints, and of how those validations can most effectively those validations can most effectively be carried outbe carried out

ReferencesReferencesPart IPart ITrain, K. (2003) Chapter 10 Simulation Assisted Train, K. (2003) Chapter 10 Simulation Assisted

Estimation, In “Discrete Choice Methods with Estimation, In “Discrete Choice Methods with Simulations”, Cambridge University Press.Simulations”, Cambridge University Press.

Part II Part II Abraham, J. (2000) “Parameter Estimation in Urban Abraham, J. (2000) “Parameter Estimation in Urban

Models: Theory and Application to a Land Use Models: Theory and Application to a Land Use Transportation Interaction Model of the Sacramento, Transportation Interaction Model of the Sacramento, California Region” Doctor of Philosophy Dissertation. California Region” Doctor of Philosophy Dissertation. Department of Civil Engineering, University of CalgaryDepartment of Civil Engineering, University of Calgary

Part III Part III Mutton, Sutherland and Weeks (2000) Validation of Mutton, Sutherland and Weeks (2000) Validation of

longitudinal dynamic microsimulation models: longitudinal dynamic microsimulation models: experience with CORSIM and DYNACAN, In experience with CORSIM and DYNACAN, In “Microsimulation Modelling for Policy Analysis”, “Microsimulation Modelling for Policy Analysis”, Cambridge University Press.Cambridge University Press.

Presented by: Flavia Tsang M.A.Sc Student 12 th March 2004

Documents

Transcript of Presented by: Flavia Tsang M.A.Sc Student 12 th March 2004