SENSIBLE VALIDATION FOR IW SIMULATIONS
description
Transcript of SENSIBLE VALIDATION FOR IW SIMULATIONS
![Page 1: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/1.jpg)
![Page 2: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/2.jpg)
SENSIBLE VALIDATION FOR IW SIMULATIONS
...with special attention paid to models displaying emergent behavior, and those
used to support analysis.
Michael P. BaileyOperations Analysis Division, USMC
![Page 3: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/3.jpg)
“All models are wrong, some are useful.”
George Box
Wartime Statistician
![Page 4: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/4.jpg)
WORKABLE DEFINITIONS
• Conceptual Model – Description of the system in some abstracted/symbolic formalism, usually mathematics.
• Verification – The simulation executable faithfully reflects the Conceptual Model
• Validation – The degree to which the system described in the conceptual model is appropriate in supporting the intended use.
• Accreditation – The judgement, made by someone responsible for the outcome, that the simulation is adequately verified and valid for the intended use.
![Page 5: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/5.jpg)
COMMENTS
• Conceptual models are always incomplete.• Verification of a simulation is a scientific
endeavor if the conceptual model is complete.• A simulation is never “Valid.”• Analytical intended uses are difficult to deal
with…– Repetition is very rare.– Analysts have no way to scientifically express the
intended use.
![Page 6: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/6.jpg)
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
![Page 7: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/7.jpg)
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
![Page 8: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/8.jpg)
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
Driven by analytictask. More later...
![Page 9: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/9.jpg)
executablecode
ideal sim
naturalsystem
REALITY FORBIG-IRON SIMS
abstraction
data development
![Page 10: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/10.jpg)
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal simFOR A GIVEN ANALYSIS
naturalsystem
FOR ANOTHERDAY
FO
CU
S
![Page 11: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/11.jpg)
ANALYSIS
• Predict the response (absolute)• Predict the response (as compared to a baseline)• Predict the functional form of the response for a set of
independent variables• Predict the sign of the gradient (set of 1st derivatives)• Is there any response?• Predict the min/max of the response over a high-
dimensional domain• Predict xi in [Li, Ui] such that response > c• Characterize the probabilistic nature of the response
![Page 12: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/12.jpg)
ISAAC: A ScreenshotISAAC is an agent-based simulation for ground combatCNA developed for Project Albert
![Page 13: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/13.jpg)
DISCUSSION EXAMPLE: ISAAC
• Sensor range – maximum range at which the ISAACA can sense other ISAACAs
• Fire range – area within which the ISAACA can engage enemy ISAACAs in combat (also allows for fratricide)
• Threshold range – area within which an ISAACA computes the numbers of friendly and enemy ISAACAs that play a role in determining what move to make on a given time step
• Movement Range – area defining where an ISAACA can select a move from on a given time step (1 or 2)
• Communications Range – boxed area within which friendly ISAACAs can communicate the information of their local sensor fields to the ISAACA
![Page 14: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/14.jpg)
Move selection
For every possible move an ISAACA can make, a penalty is calculated using the personality weight vector. ISAACA moves to the square that best satisfies personality-driven “desire” to get closer to (or farther away from) friendly and enemy ISAACAs and enemy (or own) goal.
![Page 15: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/15.jpg)
ISAACA Combat• Each ISAACA given opportunity to
fire at all enemy ISAACAs within range, unless constraint in place*
• If an ISAACA is shot, its current state is degraded from alive to injured or from injured to dead
• Probability that a given enemy ISAACA is shot is fixed by user specified single-shot probabilities
• Probability for an injured ISAACA is equal to one half of its single-shot probability when it is alive
![Page 16: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/16.jpg)
s1
Courtesy of Project Albert
![Page 17: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/17.jpg)
s1
Courtesy of Project Albert
![Page 18: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/18.jpg)
s1
Courtesy of Project Albert
![Page 19: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/19.jpg)
s2
Courtesy of Project Albert
![Page 20: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/20.jpg)
SOME TERMS WE USE
• Distillation: Representing a physical process using a simple, controllable model– e.g. Range rings for radar detections, the
ISAACA movement (direction and speed) model
• Emergent Behavior: Group behavior of entities that appears to be organized, but is generated from distributed individual logic– e.g. Encirclement
![Page 21: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/21.jpg)
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem•Simulations displaying
emergent behavior are difficult to validate because it is difficult to predict their behavior from the Conceptual Model
•Therefore there is greater pressure to use results validation.
![Page 22: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/22.jpg)
"The more complex the model, the harder it is to distinguish unusual
emergent behavior from programming bugs."
Douglas Samuelson
Renowned Operations
Research Analyst
![Page 23: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/23.jpg)
TECHNICAL SENSITIVITIES
• Activation pattern
• Simulation language
• More?
The response should NOT be sensitive to these.
![Page 24: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/24.jpg)
Schism• Agent-based simulations use modular rules
and local reasoning to produce realistic and/or interesting emergent aggregate behavior.– Surprise is good**
• Successful simulation testing (core to face/results validation) based on demonstrating credibility across the range of potential input.– Surprise not good**
** Refined later in this talk
![Page 25: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/25.jpg)
GOAL: STOP BEING SURPRIZED
Surprise
Explore
Explain
Accept/reject
ProductionRuns
How do w
e te
ll ab
out this
exper
ience
?
In control,no more surprises
1. “Unnatural acts” reflect negatively on a sim
2. Once we chieve top-down control, is there still emergent behavior?
![Page 26: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/26.jpg)
TERMS
• Spreadsheet model
• Object-oriented
• Agent-based
These are terms pertaining to software design, and are irrelevant to the utility of the resulting software insupporting analysis.
![Page 27: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/27.jpg)
SIMULATION-SUPPORTED ANALYSIS
• Baseline/Excursion or Factorial Experiment
• Driven to answer Analysis Questions
• Key Elements of Analysis
• Constraints, Limitations, and Assumptions
![Page 28: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/28.jpg)
ELEMENTS
SIMULATION DYNAMICS• based on accepted physical
laws• based on accepted social
dynamics• based on common sense• distillation
– simple model relic required to facilitate actions
– simple model relic required to maintain consistency
• top-down human intervention
DATA• authoritative value• measured• witnessed• argued by logic• sensible range• guess/arbitrary• dimensionless
RELEVANT DYNAMICS + REQUIRED DATA = ELEMENTe.g. underwater detection
using observed detection range data in a cookie-cutter model
GO
OD
NE
SS
![Page 29: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/29.jpg)
RELEVANT ELEMENTS
CORE ELEMENTS: drive the results of your experiment, align with the keyelements of analysis
INTERESTING ELEMENTS: have impact on the results, but are not elements of analysis
NECESSARY ELEMENTS: necessary tosupport the model, give the analysis context
CLA: Constraints, Limitations, and Assumptionsnecessary to give context, scope the analysis,and interpret the results1
1An excellent FOUO document exists on this subject. Seeking unlimited release. Watch orsagouge.pbwiki.com.
![Page 30: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/30.jpg)
CORE
• Drive the results, dominate the effect of other elements
• Pertain directly to the goals of the analysis
• Are the key elements in the experiment
![Page 31: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/31.jpg)
INTERESTING
• Influential elements not directly connected to goals of the analysis
• Sources of variability in context
• May comprise cases within a baseline or excursion
![Page 32: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/32.jpg)
NECESSARY
• Required to facilitate the simulation
• Can provide a variety of contexts weighted by their plausible frequency
• Believed not influential, often fixed
• Give the outcome meaning, interpretability
![Page 33: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/33.jpg)
CLA
• Important context, scoping, or caveats that must accompany the analysis results
• Believed to be valid for the purposes of the analysis
• Knowledge gained can be extrapolated beyond the CLA with caution
• Stated a priori, and agreed to by analysis sponsor prior to commencement
![Page 34: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/34.jpg)
EXAMPLE
• ANALYSIS GOAL: Determine the relative effectiveness of USN fleet sizes for aviation-capable amphibious ships
• MOE’s– JFEO – time to deliver Assault Echelon– SASO -- % deck space and aircraft available for non-routine
tasking• CORE: number of ships in the fleet (implied -- closure
times and embarked aircraft)• INTERESTING: MAGTF ACE composition, temperature,
posture of units ashore• NECESSARY: terrain at LZ’s, order of lift, serial loading,
tasking• CLA: scenario, Joint TACAIR support
![Page 35: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/35.jpg)
IMPACT ON ANALYSIS
• Agent-based design is reputed to enable fast and easy construction of interesting and necessary elements
• Simulation environments usually provide pre-programmed interesting and necessary elements, and plug-in capability for core elements
• Necessary and Interesting elements can display emergent behavior to add variability
– Emergent behavior is often not predictable/controllable• Big-iron simulations often have parametric (knob) control
over necessary elements– impossible to promote these to interesting or core elements– should NOT be elements of analysis
• Ideally, analysts should have the most faith in their core elements
– should have high-quality data (high on the GOODNESS scale)– should have well-studied dynamics (high on the GOODNESS scale)– must not display emergent behavior
• Core elements should be results-proven to be highly influential (see Scientific Method of Choosing Model Fidelity)
• Limitations on the Core = Limitation of the simulation for analytical purposes
• Core and Interesting Elements should results-proven to be consistent with SME (explainable 1st derivative)
![Page 36: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/36.jpg)
NEGATIVE INFORMATION for IN-VALIDATION• Elements not data-driven• Elements not controllable• Element displays undesired emergent
behavior• Element displays unexplainable 1st-order
influence (results schism unexplainable)• Element not in the anticipated layer
– level of influence is more/less than anticipated by analyst
– dynamics or data are...• too low on the GOODNESS scale• mismatched vis. the GOODNESS scale
![Page 37: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/37.jpg)
NEGATIVE INFORMATION = IN-VALIDATION?
no concern show stopper
Negative information about an element may/may not be a show-stopper1. influence (Core-ness) of the element vis. the analysis2. harshness of the negative information
![Page 38: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/38.jpg)
“Computer programs should be verified,Models should be validated,
and Analysts should be accredited.”
Alfred G. BrandsteinRenowned Military Operations Research Analyst
Founder of Project Albert
![Page 39: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/39.jpg)
THE ANALYST• Prior to any experience with the simulation, can the Analyst...
– Pose analytic questions mathematically?– Describe the experiment?– Identify core & interesting elements?– Specify CLA elements?– Evaluate core elements on “good-ness” scale?– Disclose all outcomes the analyst anticipates matching with the simulation (Test
Cases)?• Once experience has been gained, can the Analyst...
– Explain changes to anticipated core/interesting/necessary classification?– Describe all experimentation on distilled elements?– Describe all testing and tuning required?– Quantify level of influence of each core & interesting element statistically?– Explain the impact of each CLA on the results?– Statistically determine the level of agreement of the simulation outcomes with the
Test Cases?• Resulting analysis should be peer-reviewed
![Page 40: SENSIBLE VALIDATION FOR IW SIMULATIONS](https://reader036.fdocuments.us/reader036/viewer/2022062422/5681400e550346895dab493a/html5/thumbnails/40.jpg)
Bottom line -- if the analyst cannot describe his application and predict certain things, the model/application
match is doomed from the start.