Post on 01-Apr-2015
2009, Aptima, Inc. 1
www.aptima.comMA ▪ DC ▪ OH ▪ FL
© 2009, Aptima, Inc.
Human-Centered Engineering Perspectives on Simulation-Based Training
Daniel SerfatyEmily Wiese
Presented to SimTransCopenhagen, Denmark22 June 2009
2009, Aptima, Inc. 2
Agenda
Introduction to Aptima– Examples of Capabilities in Human-Centered Engineering
Five Emerging Technologies in Simulation-Based Training– Scenario Engineering– Simulation Fidelity– Performance Measurement (with A-Measure toolkit demo)– Cognitive Skills Training– Team Communications Assessment
Discussion
2009, Aptima, Inc. 3
What is Human–Centered Engineering?
TechnologyCapabilities
Social &Organizational
Structures
Mission, Tasks & Work
Processes
Human Agents
Congruence
2009, Aptima, Inc. 44
Aptima, Inc.
Interdisciplinary Small Business – Founded in 1995– Consistent annual growth (40% CAGR)– 100+ staff (80% graduate degrees)
Human Centered Engineering– Analyze and design complex socio-technical
systems– Combine social science theory with
quantitative, computational methods Serving government and commercial clients
– 350+ contracts with the Defense Industry Offices
– Boston/Woburn, MA, (HQ) – Washington, DC – Dayton, OH– Ft Walton Beach, FL
2009, Aptima, Inc. 55
Domain Expertise
Skill Set
Command & Control Military Training Leadership Complex Information Display National Security Solutions Medical & Healthcare Aviation Emergency Preparedness Stability & Support Operations Education Safety
Educational Backgrounds
2009, Aptima, Inc. 6
Optimizing Performance in Mission-Critical Environments
2009, Aptima, Inc. 7
Examples of Capabilities
Performance Measurement Socio-Cultural Applications Organizational Engineering Training
2009, Aptima, Inc. 8
2009, Aptima, Inc. 9
2009, Aptima, Inc. 10
Official U.S. Navy photo. Neither the U.S. Navy nor any other component nor any other component of the Department of Defense has approved, endorsed, or authorized this product [or promotion, or service, or activity].
2009, Aptima, Inc. 11
2009, Aptima, Inc. 12
2009, Aptima, Inc. 13
2009, Aptima, Inc. 14
2009, Aptima, Inc. 15
2009, Aptima, Inc. 16
2009, Aptima, Inc. 17
2009, Aptima, Inc. 18
2009, Aptima, Inc. 19
2009, Aptima, Inc. 20
2009, Aptima, Inc. 21
Agenda
Introduction to Aptima– Examples of Capabilities in Human-Centered Engineering
Five Emerging Technologies in Simulation-Based Training– Scenario Engineering– Simulation Fidelity– Performance Measurement (with A-Measure toolkit demo)– Cognitive Skills Training– Team Communications Assessment
Discussion
The New Science of Scenario Engineering
BEST– Engineering the Stimulus for Optimal Learning
PRESTO– Optimizing Learning Trajectories Using Constraint-Based Logic
CROSSTAFF– Engineering Training Scenarios from Operational Data
VSG– DDD Visual Scenario Generator Tool
Q: How to Optimize Learning on a Given Simulator?
Vision: Learning from Synthetic Experiences
Training Opportuniti
es
Training Experienc
esNext Scenario
Readiness
Competencies, Knowledge, Skills…Training
Objectives
Phase I
0
4
8
12
16
20
flag alpha bravo charlie
Cell
Worklo
ad
selfrating
othersrating
Phase II
0
4
8
12
16
20
flag alpha bravo charlie
Cell
Worklo
ad
selfrating
othersrating
What to Measure
Performance Measurement
Data
BEST: Optimizing Scenario Selection
Assess team performance against near-optimal solution Based on that assessment, select training events to
optimize the team’s learning curve
R2 = 0.966 R2 = 0.9841
41000
42000
43000
44000
45000
46000
47000
48000
Mission
Te
am
De
fen
siv
e S
co
re
MostControlMost TrendlineControl Trendline
Training Session
Tea
m S
core
What sequence of experiences moves a team to a steeper learning curve?
System’s Belief about State of Competencies
True State of Competencies
s(n-1)s(n-1) s(n)s(n) s(n+1)s(n+1)
X(n-1)X(n-1) X(n)X(n) X(n+1)X(n+1)
scenarioat n
scenarioat n
rewardat n
rewardat n
observationat n
observationat n
scenarioat n+1
scenarioat n+1
rewardat n+1
rewardat n+1
Parameters set by SMEs
Parameters set by SMEs
Tool: Partially Observable Markov Decision Process (POMDP) Training Model
Experiment Result:52% Improvement over Best
Practices Team Training
Conventional vs. PRESTO-Based Scenarios
Conventional Scenario MSEL EventsPlanned Actually Occurred
Didn’t Occur
PRESTO-based ScenarioActual
Actual
PRESTO-Based Scenario Space
(a) Original Flight Path (b) Key Events Identified
(d) Scenario Envelope Generated(c) Events Generalized
Engineering Training Scenarios from Operational Data (CROSSTAFF)
Scenario Engineering Using the DDD* Visual Scenario Generator (VSG)
DDD*: Distributed Dynamic Decision-making simulator
Pedagogically-focusedAdaptive Scenarios
Scenario
Initial
Conditions
Participant
Advice
Exercise
Events
Training Objectives
TO Conditions
Ongoing TrainingExercise
PRESTO/CROSSTAFF
SAF/Instructor
Advice
Performance Measurement
System
(A-Measure)
Trainee Performance
Measures
BEST Optimal Vignette
Sequencing
Understanding Simulator Fidelity Requirements
1. There is little guidance and no standard tool for determining the appropriate level of fidelity of training simulators to – Achieve specified training
objectives,– Maintain trainee acceptance,
and – Fit within budgetary constraints.
Simulator Fidelity
Tra
inin
g E
ffect
ive
nes
s
2. There are no standard measures designed to be sensitive enough to detect objective performance differences invoked by varying levels of fidelity
Perceived
Act
ual
Co
st
2009, Aptima, Inc.
31
RELATE: A Research-Driven Approach
A systematic approach to establish quantitative, predictive relationships between simulator fidelity and training effectiveness
RELATE fuses:– Fidelity requirements defined by end-users;– Existing theory and research about fidelity; and – Objective performance data from fidelity experiments to develop a
predictive, computational model.
2009, Aptima, Inc.
2009, Aptima, Inc. 32
Model-Based Tool
A computational model-based tool to assist with decisions regarding the acquisition and use of training simulators
Tool can help users: Conduct return on investment
analyses to determine which simulator to develop or acquire
Prioritize technology enhancements to improve the effectiveness of existing simulators
Develop an strategy for employing both high- and low-fidelity simulators to meet training objectives
2007, Aptima, Inc. 33
PERFORM: Air-Air Combat Fidelity Requirements
Conducted research to examine the visual and cockpit fidelity requirements for training air-to-air combat skills to experienced pilots in F-16 simulators
Developed a decision-support tool to help the Air Force prioritize technology enhancements to improve the effectiveness of deployable simulators
Deployable Tactics Trainer (DTT)Display for Advanced Research and Technology (DART)
34
PREDICT: Air-Ground CombatFidelity Requirements
Conducting research to examine the visual fidelity requirements for training air-to-ground combat skills to inexperienced pilots in F-18 simulators
Developing a decision-support tool to help the Navy develop a strategy for employing both high- and low-fidelity simulators to meet training objectives
2009, Aptima, Inc.
35
FLEET: Developing and ValidatingFidelity-Sensitive Measures
Developing measures of pilot performance that are sensitive enough to detect objective performance differences invoked by varying levels of fidelity
2009, Aptima, Inc.
Fidelity-Sensitive Performance Measures
Does the pilot complete Combat Fence IN Checks?
Does the pilot maintain briefed formation?
Does the pilot appropriately mitigate surface-to-air threats?
Does the pilot deconflict with other assets?
Conducting research to examine the motion fidelity requirements for training air-to-ground combat skills to inexperienced pilots in T-45 simulators
Summary
Aptima seeks to identify the simulator fidelity requirements for effective training by: Developing a systematic approach to establish quantitative,
predictive relationships between simulator fidelity and training effectiveness
Building a computational model-based tool to predict the impact of simulator fidelity on training effectiveness
Creating performance measures and measurement tools that can be used to collect better data in simulator fidelity experiments
Employing the proper level of fidelity will ensure better training results and reduce costs by eliminating investments in unnecessary training and technology
2009, Aptima, Inc.
New Approaches to Measuring Performance
A well-designed measurement system makes simulation-based practice effective training– The right feedback to the right person at the right time leads to better
learning
Measurement enables assessment of training effectiveness – Are people getting the skills they need?
Guides selection of training environment– Live, virtual, constructive
– Facilitates appropriate use of measures
Measurement technology can turn simulators into training machines
You Can’t Train What You Can’t Measure
Why is this hard?– Volume of data
– “We recorded everything...”
– Lack of meaningful aggregation methods– “325,435 messages were received…the average length was 2.35 minutes”
– Interdependence of behaviors at different locations– No one person has the total picture
– “Correct” behavior depends on dynamic context– It is hard to construct, even after the fact, where the team went wrong
Hours spent training ≠ Proficiency
Real-world experience ≠ Proficiency
Performance Measurement Process
Competency Based Performance Measurement
Competency-based performance measures
Leverages performance measurement theory in combination with subject matter expert input
Assesses team and individual performance
The COMPASSSM
Methodology is a product of Aptima, Inc.
COMPASSSM tells us the what direction to go with measure development
A▪Measure Product Family
Turning simulators into training machines
Cognitive Skills Training
Not all skills should be trained the same way– Cognitive skills vs. procedural skills
Pedagogical Theory: Direct instruction vs. constructivism Most types of training/education employ a direct instruction
approach– Can be effective for training procedural skills
Current research suggests that a problem-based approach may be more appropriate for training cognitive skills– What differs is when you tell someone how to do something relative to
when they practice doing it E.g., telling students the formula for density and having them practice (the traditional
approach) enforces a plug-n-play understanding. Very little transfer. Instead, give students a situation and ask them how they would describe density.
– Now they get a sense of the principles involved before you give the formula.
The Bransford Model
Community-Centered(Relevance)
Assessment-Centered(Formative Feedback)
Knowledge-Centered(Deep Understanding)
Learner-Centered(Where do Learners Start)
Aptima’s Balanced Unified Incremental Learning Development (BUILD)
Training Approach
Example: How to “train the ear”
Air Battle Managers (ABMs) monitor multiple communications (radio) channels at the same time
This currently an acquired skill developed through experience and on the job training.
How can we train a novice?
At its core, this is a skill that relies heavily on cognitive skills like memory and attention.
Monitoring multiple communications channels requires…– Dedicating limited cognitive resources (memory)
– To attending to stimuli that must pass a certain threshold (attention)
– Under stressful conditions (stress)
How can these cognitive concepts help ABMs?
“Monitoring multiple comms channels requires…– Dedicating limited cognitive resources (memory)”
Psychological research demonstrates that one can free limited working memory resources by placing some information in long term memory.
The key to long term memory storage is automaticity.– When I see X, I do Y
The key to training automaticity is repetition– Multiple trials
How can these cognitive concepts help ABMs?
“Monitoring multiple comms channels requires…– Dedicating limited cognitive resources (memory)”
– To attending to stimuli that must pass a certain threshold (attention)”
The threshold is often physical (a certain volume, a certain brightness), but could also be semantic (meaningful).– Cocktail party effect
Trigger words can break the semantic threshold for attention– When I hear an important word, I attend to it
This is the mechanism for retrieval.
From Theory to Application: A Layered Comms Training Approach
Phase I: Trainees are introduced to ABM trigger words and their definitions. – Begin to store in long term memory
Phase II: Trainees recognize trigger words within a stream of communications.– Develop automaticity and retrieval
Phase III: Trainees recognize a trigger word in a realistic scenario and respond accordingly.– Automatically retrieve under stressful conditions
Cognitive Skills Training
CIFTS: Communications Analysis in Operational Environments
2009, Aptima, Inc. 50
Domain: Air and Space Operations Center (AOC)
Numerous centers around the world
Around 100 operators communicating In the air On the ground Around the world
Extremely complex operations must be coordinated
Extensive use of chat to coordinate, assign tasks, exchange information
“Communications is at the Heart of Team Performance”
CIFTS Project Domain: AOC Chat Data
2009, Aptima, Inc. 51
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position standby dtl updated[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TDO> TGTS: please might mean[23:12] <TGTS> sir bent no[23:13] <SOLE> high fast flyer affirm[23:13] <TGTS> c
[23:09] <GTC> still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> GTC: relay inform pol leadership lost[23:10] <GTC> current position standby
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position standby dtl updated[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TDO> TGTS: please might mean[23:12] <TGTS> sir bent no[23:13] <SOLE> high fast flyer affirm[23:13] <TGTS> c[23:14] <CCO> Chief: attack check jdocs sado right wrkg unfriendly lost track[23:14] <GTC> SOLE: anyone did you copy can you confirm[23:15] <SADO> ATK: what up[23:16] <GTC> still issues[23:16] <TGTS> TDO: wrking wrong sam ring return to base facilities who is disgard[23:16] <SADO> SODO: jadocs each time wrong voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <GTC> jstars armored car link track[23:17] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> resend[23:21] <GTC> not good wmd vehicles where[23:22] <SOLE> wmd vehicles ukn roll call requested do u agree[23:23] <SADO> GTC: all c2 players do u launch pads rqist control measure was back up[23:23] <CCO> no idea[23:25] <Chief> SADO: strike asset changing cco[23:25] <SODO> negs cvy need no pred[23:25] <SOLE> what were you doesnt vehicle assembly msn results[23:25] <SIDO> ATK: got it mass of vehicles pimp
Need automated methods to understand what is happening in general, and the ability to drill down to specific instances
Advanced Language Analysis:LAVA TOOLkit
LAVA is Aptima’s LAtent Variable Analysis toolkit LAVA provides tools for natural language processing
– Processing free text to represent words as numbers– Statistical tools to extract concepts from free text
LAVA is language-independent– Inter-agency, inter-cultural, international differences
The kinds of questions you can ask of LAVA– What are the main concepts within this set of news articles?– How similar are these two medical records?– What is this e-mail talking about?– How does the subject change over time?
LAVA in JAVA– Intel Math Kernel Library– Microsoft SQL Server or MySQL– Web services– JAVA API
52 2009, Aptima, Inc.
PACE: Processing and Analysis of Communications and Events
Addressee Density
– High Density Message Type
– Question– Acknowledgment– Command– Ambiguity– Pause
Valence– Positive– Negative
Mission ID– JA0002 == Underground Bunker
Chains– GTC – TDO
Combinations….
2009, Aptima, Inc. 53
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn
Non-Combatant Evacuation Data Example: Valence and Type
2009, Aptima, Inc. 54
CIFTS Timeline Interface
2009, Aptima, Inc. 55
Tick marks indicate messages in timeline. Colors indicate
values for the current analysis.
Any two analyses can be “crossed”
Messages can be filtered by
checking values on each analysis type
Tabs display different analyses
Content of messages
Tabs hold different kinds of summary data
CIFTS Network Interface
2009, Aptima, Inc. 56
Arrow A->B indicates message from A was followed by a message from B within the same chat room; on average, a “response”
Arrow thickness indicates % of messages from A that were followed by messages from B; “conversations”
Size of the circle indicates percent of all messages that this member sent
Length of the line indicates the time interval between messages; “density” of comms
Layout tries to minimize line-crossings; localizes “functionality”
More “central” members have contacts with more other members
Conclusions & Questions
Key Questions:– In a world dominated by simulation-based training,
how do we optimize learning yield (ROI)?– In scenario-based training, what is the “curriculum”?– How can we get effective and efficient in performance
measures selection and feedback?– How can rigorous scientific methods help contribute to
the above?– Scientific methods Practical software tools?
Daniel Serfaty: serfaty@aptma.com
Emily Wiese: ewiese@aptima.com