1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display...
-
Upload
ethelbert-stevenson -
Category
Documents
-
view
219 -
download
0
Transcript of 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display...
![Page 1: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/1.jpg)
1
ABSVal
Goal: Adapt accepted Best Practices of VV&A* to simulation models that...
a. Display emergent behavior
b. Are used to model military effects on population dynamics and social phenomena as well as military decision making
c. Support analyses*maybe refine the general Best Practices
![Page 2: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/2.jpg)
2
B.L.U.F.WHAT I THINK WE LEARNED or CONFIRMED
• VV&A universally reviled.– EXCEPTION: DMSO employees and alumni
• VV&A general principles do not map well to analysis domain.– describing anticipated use is problematic– analysis = exploration
• ABS translates to Models exhibiting Emergent Behavior.– pre-production experimentation focused on achieving top-down control (building predictive
capability) on dynamics that initially display emergence– scientifically examining emergence is interesting
• We can help analysts and decision makers distinguish good analysis from bad analysis.
• Minimally acceptable sim-to-app validation is recognizable, achievable, useful, and rare.
![Page 3: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/3.jpg)
3
OUTLINE1. SIMULATION
MODELS
2. A LITTLE EMERGENT BEHAVIOR
3. THINKING ANALYSIS
4. EXPOSING THE MATCH
![Page 4: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/4.jpg)
4
“In contrast to this interest in model-related technology, therehas been far too little interest in the substance of the models and the validity of the lessons learned from using them. In ourview, the DoD does not appreciate that in many cases themodels are built on a base of sand.”
The Base of Sand Problem: A White Paper on the State of Military Combat Modeling.Paul K. DavisDonald Blumenthal
![Page 5: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/5.jpg)
5
WORKABLE DEFINITIONS
• Conceptual Model – Description of the system in some abstracted/symbolic formalism, usually mathematics.
• Verification – The simulation executable faithfully reflects the Conceptual Model
• Validation – The degree to which the system described in the conceptual model is appropriate in supporting the intended use.
• Accreditation – The judgement, made by someone responsible for the outcome, that the simulation is adequately verified and valid for the intended use.
![Page 6: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/6.jpg)
6
COMMENTS
• Conceptual models are always incomplete.
• Verification of a simulation is a scientific endeavor if the conceptual model is complete.
• A simulation is never “Valid.”
• Analytical intended uses are difficult to deal with…– Repetition is very rare.– Analysts have no way to scientifically express the intended use.– Analysts often accept very poor data and models, and often
express grave caveats for their results.
![Page 7: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/7.jpg)
7
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
![Page 8: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/8.jpg)
8
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
![Page 9: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/9.jpg)
9
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
Driven by analytictask. More later...
![Page 10: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/10.jpg)
10
executablecode
ideal sim
naturalsystem
REALITY FORBIG-IRON SIMS
abstraction
data development
![Page 11: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/11.jpg)
11
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal simFOR A GIVEN ANALYSIS
naturalsystem
FOR ANOTHERDAY
FO
CU
S
![Page 12: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/12.jpg)
12
"The more complex the model, the harder it is to distinguish unusual
emergent behavior from programming bugs."
Douglas Samuelson
Renowned Operations
Research Analyst
![Page 13: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/13.jpg)
13
ABSVal PROJECT
• ABSVal Framework
• Test Examples– Pythagoras COIN– SZ/BZ Obstacle Reduction Analysis
• Conclusions
![Page 14: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/14.jpg)
14
VALIDATION
• First-Principles Validation– Assess the theory behind
the conceptual model• And predict its impact on
the ensuing analysis– Examine the
implementation of the theory
• And predict its impact on the ensuing analysis
– Examine the combinations of theories used together
• Results Validation– Compare output data to
data from another source• Historical case• Another model• Intuition
![Page 15: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/15.jpg)
15
VALIDATION
• First-Principles Validation– Assess the theory behind
the conceptual model• And predict its impact on
the ensuing analysis– Examine the
implementation of the theory
• And predict its impact on the ensuing analysis
– Examine the combinations of theories used together
• Results Validation– Compare output data to
data from another source• Historical case• Another model• Intuition
VALIDATION = EXPOSITION + ASSESSMENTThe EVIDENCE on which acceptance is based.
![Page 16: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/16.jpg)
16
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem•Simulations displaying
emergent behavior are difficult to validate because it is difficult to predict their behavior from the Conceptual Model
•Therefore there is greater pressure to use results validation.
![Page 17: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/17.jpg)
17
“All models are wrong, some are useful.”
George Box
Wartime Statistician
![Page 18: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/18.jpg)
18
ANALYSIS
• Predict the response (absolute)• Predict the response (as compared to a baseline)• Predict the functional form of the response for a set of
independent variables• Predict the sign of the gradient (set of 1st derivatives)• Is there any response?• Predict the min/max of the response over a high-
dimensional domain• Predict xi in [Li, Ui] such that response > c• Characterize the probabilistic nature of the response
Compare to physical sciences -- These are very humble goals.Might a medical/biological mindset be more appropriate?
![Page 19: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/19.jpg)
19
…more ANALYSIS
Provide the best decision support possible, and include useful assessment of the value of the analysis vis. the questions & issues at hand.
![Page 20: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/20.jpg)
20
IDEAL STUDY PROCESS
DETERMINETHE QUESTION
DETERMINETHE QUESTION
DETERMINETHE MOE’s and the EEA’s
DETERMINETHE MOE’s and the EEA’s
FIND or BUILDthe BEST (SIMULATION)MODEL
FIND or BUILDthe BEST (SIMULATION)MODEL
PRODUCTIONRUNS
PRODUCTIONRUNS
PRESENT (and DEFEND)RESULTS
PRESENT (and DEFEND)RESULTS
![Page 21: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/21.jpg)
21
SIMULATION-SUPPORTED ANALYSIS
• Baseline/Excursion or Factorial Experiment
• Driven to answer Analysis Questions
• Key Elements of Analysis
• Constraints, Limitations, and Assumptions
![Page 22: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/22.jpg)
22
Schism• Agent-based simulations use modular rules
and local reasoning to produce realistic and/or interesting emergent aggregate behavior.– Surprise is good**
• Successful simulation testing (core to face/results validation) based on demonstrating credibility across the range of potential input.– Surprise not good**
** Refined later in this talk
![Page 23: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/23.jpg)
23
GOAL: STOP BEING SURPRIZED
Surprise
Explore
Explain
Accept/reject
ProductionRuns
How do w
e te
ll ab
out this
exper
ience
?
In control,no more surprises
1. “Unnatural acts” reflect negatively on a sim
2. Once we achieve top-down control, is there still emergent behavior?
![Page 24: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/24.jpg)
24
ELEMENTSAdaptation of the Yost Scale
SIMULATION DYNAMICS• based on accepted physical
laws• based on accepted social
dynamics• based on common sense• distillation
– simple model relic required to facilitate actions
– simple model relic required to maintain consistency
• top-down human intervention
DATA• authoritative value• measured• witnessed• argued by logic• sensible range• guess/arbitrary• dimensionless
RELEVANT DYNAMICS + REQUIRED DATA = ELEMENTe.g. underwater detection
using observed detection range data in a cookie-cutter model
![Page 25: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/25.jpg)
25
ELEMENTSAdaptation of the Yost Scale
SIMULATION DYNAMICS• based on accepted physical
laws• based on accepted social
dynamics• based on common sense• distillation
– simple model relic required to facilitate actions
– simple model relic required to maintain consistency
• top-down human intervention
DATA• authoritative value• measured• witnessed• argued by logic• sensible range• guess/arbitrary• dimensionless
RELEVANT DYNAMICS + REQUIRED DATA = ELEMENTe.g. underwater detection
using observed detection range data in a cookie-cutter model
CO
NT
RO
LA
BL
E A
BS
TR
AC
TIO
NA
NA
LY
TIC
AL
LY
DE
SIR
AB
LE
![Page 26: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/26.jpg)
26
“It’s the Data, Stupid.”Phalanx, DEC 07George Akst
![Page 27: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/27.jpg)
27
Constraints, Limitations, and Assumptions Guide
TRADOC Analysis Center255 Sedgwick Avenue
Fort Leavenworth, KS 66027-2345
TRAC-TD-05-011 (rev. 1)January 2008
Mike Bauman
![Page 28: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/28.jpg)
28
PARSING SOURCES OF VARIABILITY
CORE: drive the results of your experiment, align with the keyelements of analysis
DYNAMIC CONTEXT: has impact on the circumstances relevant to exercising the core model dynamics, create situations, not elements of analysis
CASES: details necessary tosupport the model, cases to be consideredto achieve analytical goals
C.L.A.: Constraints, Limitations, and Assumptionsnecessary to give scope the analysis,and interpret the results
![Page 29: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/29.jpg)
29
IMPACT ON ANALYSIS
• Agent-based design is reputed to enable fast and easy construction of dynamic context
• Dynamic Context elements can display emergent behavior to add variability
– Emergent behavior is often not predictable/controllable• Big-iron simulations often have parametric (knob) control
over Case elements– impossible to promote these to Dynamic Context or Core
elements– should NOT be elements of analysis
• Ideally, analysts should have the most faith in their Core elements
– should have high-quality data (high on the YOST scale)– should have well-studied dynamics (high on the YOST scale)– must not display uncontrolled emergent behavior
• Limitations on the Core = Limitation of the simulation for analytical purposes
• Core and Dynamic Context Elements should results-proven to be consistent with SME (explainable 1st derivative)
• Core elements should be results-proven to be highly influential (see Scientific Method of Choosing Model Fidelity)
![Page 30: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/30.jpg)
30
IMPACT ON ANALYSIS• Agent-based design is reputed to enable fast and easy construction
of dynamic context• Dynamic Context elements can display emergent behavior to add
variability– Emergent behavior is often not predictable/controllable
• Big-iron simulations often have parametric (knob) control over Case elements
– impossible to promote these to Dynamic Context or Core elements– should NOT be elements of analysis
• Ideally, analysts should have the most faith in their Core elements– should have high-quality data (high on the YOST scale)– should have well-studied dynamics (high on the YOST scale)– must not display uncontrolled emergent behavior
• Limitations on the Core = Limitation of the simulation for analytical purposes
• Core and Dynamic Context Elements should results-proven to be consistent with SME (explainable 1st derivative)
• Core elements should be results-proven to be highly influential (see Scientific Method of Choosing Model Fidelity)
Taxonomy for sources of variability reflecting the relationship between modeldynamics and analytical goals.
** Jargon for communicating how a sim element relates to the analysis.
** Identifies appropriate role for elements withemergent behavior in an analysis.
![Page 31: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/31.jpg)
31
GOLDEN GATE BRIDGE
a solid connection…
simulation/data
capabilities
analytical
requirements
![Page 32: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/32.jpg)
32
TACOMA NARROWS BRIDGE
or, not so much.
simulation/data
capabilities
analytical
requirements
![Page 33: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/33.jpg)
33
Core Experimentaln Parametric Settings
Dynamic
ContextStochastic
1 Parametric Setting
Cases Discrete Cases m Cases
CLA Static1 fixed set of assumptions
RECOMMENDED HANDLING
![Page 34: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/34.jpg)
34
EXAMPLE
• Question: What is the tactical value of LW components to a rifle squad?
• Core: weapon, computer/comm/SA, sight/NVG
• Dynamic Context: paths of maneuver, acquisitions and detections, paths & actions of threat, attrition, …
• Cases: terrain type (urban, jungle, alpine), scale (company, platoon), mission (HVT, defend a FOB)
• CLA: kinetic outcome, unambiguous threat, terrain representation
![Page 35: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/35.jpg)
35
EXAMPLE
• Question: What is the tactical value of LW components to a rifle squad?
• Core: weapon, computer/comm/SA, sight/NVG
• Dynamic Context: paths of maneuver, acquisitions and detections, paths & actions of threat, attrition, …
• Cases: terrain type (urban, jungle, alpine), scale (company, platoon), mission (HVT, defend a FOB)
• CLA: kinetic outcome, unambiguous threat, terrain representation
emergent behavior
dynamics fit here
don’t average over
these cases
![Page 36: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/36.jpg)
36
“Those claims to knowledge that are potentially falsifiable can then be admitted to the body of empirical science, and then further differentiated according to whether they are (so far) retained or indeed are actually falsified.”
Carl PopperPhilosopher of Science
![Page 37: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/37.jpg)
37
NEGATIVE INFORMATION for IN-VALIDATION• Elements not data-driven• Elements not controllable• Element displays undesired emergent
behavior• Element displays unexplainable 1st-order
influence (results schism unexplainable)• Element not in the anticipated layer
– level of influence is more/less than anticipated by analyst
– dynamics or data are...• too low on the Yost scale• mismatched vis. the Yost scale
![Page 38: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/38.jpg)
38
NEGATIVE INFORMATION = IN-VALIDATION?
no concern show stopper
Negative information: scopes analytical value of results.Analyst’s art: responsibly expand this scope.
“This approach uses a very unrealistic model of certain dynamics,but it creates adequate dynamic context to stimulate the core elementsin a way useful to our analytic goals.”
![Page 39: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/39.jpg)
39
“Computer programs should be verified,Models should be validated,
and Analysts should be accredited.”
Alfred G. BrandsteinRenowned Military Operations Research Analyst
Founder of Project Albert
![Page 40: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/40.jpg)
40
THE ANALYST• Prior to any experience with the simulation, can the Analyst...
– Pose analytic questions mathematically?– Describe the experiment?– Identify Core vs. Dynamic Context elements?– Specify CLA elements?– Evaluate Core elements on Yost scale?– Disclose all outcomes the analyst anticipates matching with the simulation (Test
Cases)?• Once experience has been gained, can the Analyst...
– Explain changes to anticipated Core/Dynamic Context/Case/CLA classification?– Describe all testing and tuning required?– Quantify level of influence of each Core & Dynamic Context statistically?– Avoid integrating (averaging) Cases?– Explain the impact of each CLA on the results?– Statistically determine the level of agreement of the simulation outcomes with the
Test Cases?• Resulting analysis should be peer-reviewed
![Page 41: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/41.jpg)
41
Bottom line
![Page 42: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/42.jpg)
42
• Have lots of computational experience with your model.
• Understand and be able to control it emergent behavior.
• Plan and execute experiments, document.
• Disclose relationship between each important sim element and the analytical goal.– Core– Dynamic Context– Cases– CLA
![Page 43: 1 ABSVal Goal: Adapt accepted Best Practices of VV&A* to simulation models that... a.Display emergent behavior b.Are used to model military effects on.](https://reader036.fdocuments.us/reader036/viewer/2022062322/5697bfb61a28abf838c9e246/html5/thumbnails/43.jpg)
43
QUESTIONS?