Post on 13-Dec-2015
MIS 643
Agent-Based Modeling and Simulation
(ABMS)
Bertan Badur
badur@boun.edu.tr
Department of
Management Information Systems
Boğaziçi University
Model Analysis
• Chapter 21-23, of Agent-Based and Individual-Based Modeling: A Practical Introduction, by S. F. Railsback and V. Grimm
Outline
• Chapter 21: Introduction to Part IV• Chapter 22: Analyzing and Understanding ABMs • Chapter 23: Sensitivity, Uncertainty and
Robustness Analysis • Chapter 24:
Chapter 21: Introduction to Part IV
• 21.1 Objectives of Part IV• 21.2 Overview of Part IV
21.1 Objectives of Part IV
• Testing – checking whether a model or submodel is correctly implemented and does what itis supposed to do
• Analysing a model: trying to understand what amodel does
• Understanding not automatic• from begining of modeling cycle
– sukbmodels or simple models– POM for sturucture, theory calibration
• Full models – frase design at some point – understand how it works and behave
• not too soon• once the model
– key processes – represent real system reasonably
• version number• two or three versions is likely• Programming and testing easy• What is science?
– relation between model and real system – POM Part III– analyse throughly – what it does
• simlfy or extend by adding new elements
• formulation few days, analysing months yearns
21.2 Overview of Part IV
• Chapter 22 • general strategies of analyzing ABMs• specific to ABMs
– structural richness and realism
• through controled simulation experiments– change assuptions submodels ...
• Chapter 23– sensitivity, uncertainty and robustness
Sensitivity, Uncertainty and Robustness Analysis
Chapter 22: Analyzing and Understanding ABMs
• 22.1 Introduction• 22.2 Example Analysis: The Segregation Model• 22.3 Additional Heuristics for Understanding
ABMs• 22.4 Statistics for Understanding• 22.5 Summary and Conclusions
22.1 Introduction
• controlled experments– varying one factor at a time – efeects on results
– establishng causal relationships – understanding how the results are affected by each factor
• Scientific method – reproducable experiments– compleatly dercribing the model - lab or field
• documenting– parameter values- input data- initial conditions
– anaylsing results of experments
• controlled simulation experiments– design, test and calibrate - models
– understanding and analyzing what models do
• How to analyse– model, the system and questions addressed,
– experience and problem solving heuristics
• Heuristics or rule of tumbs – often usefull but not always
• not unscientific •
learning objectives
• Understan purpose and goals of analyzing full AMBs– finished or preliminary
• ten heuristics• statistical anaysis for ABMs
22.2 Example Analysis: The Segregation Model
• ODD• purpose• entities, state variables and scales
– turtles – households• loaction, heppyness
– houses - patches
• space 51*51• time • stop – all heppy
• Processes• if all happy stop• far all aent• if lo limit move• update heppyness• produce output
• submodels– move
– update
Analysis
Heukristic: try extream values of parameters
• model outcomes is often easy to predict or understand
• Set tolernce low• Set tolarance high
Heuristics: findtipping points in model behavior
• qualitatively diferent behavior at extream values of parameters
• vary the parameter try to find “tipping point”– the parameter range – model behavior suddenly changes
• regiems of control– process A after some point process B may dominent
Heuristics
• try different visual representations of the model– color size patches
• run the model step by srep• look at striking or strange patterns• at interesting points keep the parmeter and vary
other parameters
22.3 Additional Heuristics for Understanding ABMs
• use several “currencies” for evaluating your simulation experiments
• analyze simplified version of your model• analyze from the buttom up• explore unrealistic senarios
Heuristics: use several “currencies” for evaluating your simulation
experiments• ABMs are rich in structure• “currincies” summary statistics or observations• emprical measures in the real system• Ex: population modeling
– measure – population size wealth
– analyze time series of population size
– even mena or range
• good currincies – observation in ODD design concept
• several currincies – how sensitive they are
• statistical distributions– mean standard deviation, range – distribution – normal, exponential
• characteristics of time series– trend, autocorrolation time units to reach a state
• measures of spatical distributions– spatial autocorrelation, fractile dimension
• measures of difference among agents– how some charcetristics different, distributions
• stability properties • network characteristics
– clustering coefficient, degree
Heuristics: analyze simplified version of your model
• simplfy• ABM so many foctors affect output• reduce complexity
– undertand what mechnizms what cause what results
• make the environment constant• make space homogenuous
– all patches same over time
• reduce stocasticity– fixed initial conditions – all agent alike– insteaad of randomness use mean values
• reduce the system size• turn off some actions in model schedule• manually create simplified initail configrations
Heuristics: analyze from the buttom up
• ABMs hard to understand • behavior of its parts – agents and their behavior• first test and undertsnd these • then full model• anaysis of submodels• developing theory for agnet bahavior
Heuristic: explore unrealistic senarios
• simulate senarios – never occur in reality• to see direct effect of a process or mechanizm on
resutls – remove it• Ex 2: How investor behavior affects double –
auction markets• interesting contrast:
– models – unrealistically simple investor behavior– produce system level results not so unrealistic
• conclusion– complex agent behavior – not reasn for complex market
dynamics– market rules themselfs might be important
22.4 Statistics for Understanding
• statistics – analysis and understanding• infer causal relatinships from a limited and fixed
data• ABM –
– generates as much data aa possible– additional mechnizms
• if cannot explain – add new mechanizms – change assuptions
• purpose and mind-set of – statistics and simulation modeling
• are quite different
• summary sttistics– aggregagting model outputs - mean, standard deviation
– extream values might be importnat so outliers are usefull
• Contrasting senarios– detect and quantify differences between senarios
– assumptions may affect resutls – number of treatments
– easier to change assuptions
– t test ANOVA
• Quantifying correlative relationships– regression ANOVA
– statistical relationsships between inputs – outputs
– inputs: paramerters, initial conditions, time series
– not directly idenfy causal relations
– but idenfity relavant factors
– meta-models
• Comparing model outputs to emprical patterns
22.5 Summary and Conclusions
• combin– reasoning, strong inference, systematic anaysis, intiution and
creativity
• once build an ABM or freeze it• understand what is does – controlled simulation
experiments• heuristics• publications• heuristics in figure 22.3• add your own
Chapter 23: Sensitivity, Uncertainty and Robustness Analysis
• 23.1 Introduction and Objectives• 23.2 Sensitivity Analysis• 23.3 Uncertainty Analysis• 23.4 Robustness Analysis• 23.5 Summary and Conclusions
23.1 Introduction and Objectives
• Does an ABM reproduce observed patterns robustly
• or sensitive to change in model– parameter
– structure
• how uncertain are model results• if model reproduce patterns foır
– parameters – limited range or values
– key processes are modelsed one exact way
• unlikely to capture real mechanizm underlying hhe patterns
Basic Definitions
• Sensitivity analysis (SA) exokıres how sensitive model’s outputs are to changes in parameter values
• Uncertainty Analysis (UA) looks at how uncertainty in parameter values affect the relaibility of model results
• Robustness analysis (RA) explores robustness ofresults and conclusions of a model to changes in its structure
Learning objectives
• local SA with BehavioSpace• visualizations – SA with several parameters or
global SA• stamdard UA methods with BehaviorSpace• steps of conducting RA
23.2 Sensitivity Analysis
• to perform SA• full version of the model• “reference” parameter set• one or two key outputs• controled simulation conditions
23.2.1 Local Sensitivity Analysis
• Objective – how sensitive the model• currency seleced• parameters one at a time• usually all parameters• Steps
– range of parameter – +or-5%
– run model for reference P and p-dP p+dp – replicate
– mean C values
– calculate sensitivity – approximatins to partial derivative
Three types of parameters
• high values of S– processes imortant in the model
• high value of S and highuncertrainty in reference valus– little information to estimate their values
– special attantion as calibration
– target of emprical research to reduce uncertainty
• low values of S– relatively unimportant processes - removable
Alternatives
• only positive change• C’/C absolugtechange• distibuton of C – variance• diferent values of P
– rgression of C on P
Limitations
• linear response so parameter change is small • parameter interractios missing• around reference parameter set
23.2.2 Analysisof Parameter Interractions via Countour Plots
• contour plots – interractions of two parameters– all other parameters are kept constant
• Multi-panel contour figures – model sensitivity – many parameters at onces
23.2.3 Global Sensitivity Analysis
• vary all parameters over their full range • look at several currencies - understanding• “brute force” - analysis
– for each parameter several values
– replicaitons
– hard to measure currencies
• regression analyis – respose surface methods• design of simulation experiments
– not all combination of parameters
23.3 Uncertainty Analysis
• similar to SA but• to understand how
– the uncertainty in parameter values and– model’s sentitivity to parameters
• interract to cause uncertainty in model results• parameters – measurment errors• steps of a UA
– identify the parameters – for each parameter – define a distribution
• belief or measurment errors
– run the modelmany times – drawing from distributions– analyze distribution of model results
23.4 Robustness Analysis
• Weisberg (2006)• Whether the results depends on the
– esentials of the model or
– details of the simplfying assuptions
• study number of distinct similar models of the same phenomena
• despte different assumptions – similar results– robust theorm - free of details of the model
• modeling, POM– robust explanations of observed patterns
• A full model – frozen• two heuristics:
– analyze simplified versions
– explore unrelistic senarios
• more complex versions• General steps of RA
– start with a well tested model version
– which elements to modify
– test modified model – reproduce observed patterns
• theory development – agent behavior– testing alternative submodels
• RA– testing alternative versions
• 23.4.1 Example: Robustness Analysis of the Breeding Synchrony Model– left as an exercise
23.5 Summary and Conclusions