1 JUP Conference, 10-11 Jan 2002 Hosted by Princeton University Quantitative Experimental Results:...
-
Upload
claude-holmes -
Category
Documents
-
view
212 -
download
0
Transcript of 1 JUP Conference, 10-11 Jan 2002 Hosted by Princeton University Quantitative Experimental Results:...
1
JUP Conference, 10-11 Jan 2002Hosted by Princeton University
Quantitative Experimental Results:
Automation to Support Time-Critical Replanning Decisions
Kip Johnson, MIT Dept. of Aero/Astro EngAdvisor Jim Kuchar, along with Dr. Charles Oman
Sponsored by Office of Naval Research
Disclaimer: The views expressed in this presentation are those of the authors and do not reflect the official policy or position of the U.S. Air Force, Department of Defense, or the U.S. Government.
2
Research Context
Time-Critical Decision-Makingie. Combat flight route planning
• Aviation, medicine, chemical and energy production, finances
Complex Problem• Unstructured aspects• Multiple competing interests and goals• Time pressures
Automation Integration• Cannot “see” everything (sensor limitations)• Human may not understand basis for automated
decisions
3
Experimental Goal
Discover the relationship between time pressures, automation assistance, and the resulting decision performance, both quantitatively and subjectively.
DecisionMaking
TimePressure
AutomationAssistance
ProblemComplexity
Human
Solution QualityRating
Quantitative&
SubjectiveScore
4
Replanning Task Description
InformationIntegration
Human
Auto
Solution
Hazards
Time toTarget
FuelConstraint
Environment:1st Order Effects
5
Experimental Protocol
4. Subject Modifies Flight Route Under Time Pressure
Minimize Threat Exposure and Time to Target Deviation
Meet Time to Target and Fuel Constraints
1. View Preplanned Mission
Finish
Start
Target
Rendezvous
Hazards
2. Change in Environment
•Hazard, Time to Target, or Fuel Update
3. Route Suggested with Varying Automation Assistance (BLUE)
New Hazard!
6
Independent Variables
Automation CategoriesNone: No Automation
original route remains
Time/Fuel: Constraint Information Only meets time to target & fuel constraints
Hazard: Hazard Information Only avoids/minimizes hazard exposure
Full: Integration of Constraint + Hazard Information minimizes hazard exposure + meets constraints
7
Independent Variables (2)
Time Pressure• 20, 28, 40, 55 seconds (logarithmic)
– capture performance change
• Unlimited time to find individual’s optimal performance
8
Dependent Variables
1. Quantitative human performance measured by route cost at end of time pressure.
Cost = Hazard Exposure (linear) + Time to Target Deviation (exponential)
Fuel = constraint
2. Subjective evaluations.
#
1 11 0
ln * * exp * 1colors
Route RouteSegment Color
tCost A Length Cost B a b
t
9
• 14 subjects: students, ave age = 25, 3 pilots, 3 females
• 3 hrs: 2 hr training, 1 hr data collection
Test Matrix:
• Greco-Latin Square Design• 4 base maps (M), each rotated for 16 effective
scenarios
Test Conditions
Time Pressure
AutoCategory
20 28 40 55 None M1 M2 M3 M4
ToT/Fuel M2 M1 M4 M3 Hazard M3 M4 M1 M2
Full M4 M3 M2 M1
10
0.0942
0.0307
-0.2526
0.1277
-0.30
-0.25
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
None ToT & Fuel Hazard Full
Automation Category
Res
idu
al S
core
Repeated Measures ANOVA
Full auto assistance is sig. best
Hazard auto assistance is sig. better than none
Automation Effects
Quantitative Analysis
IncreasingPerformance
11
0.0653
-0.0222
-0.0030
-0.0400
-0.08
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10
20 28 40 55Time (sec)
Res
idu
al S
core
• Repeated Measures
ANOVA
• No sig. performance
improvement after
28 sec
Quantitative Analysis (2)
Time Pressure Effects
IncreasingPerformance
12
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
15 25 35 45 55Time Pressure (sec)
Ab
solu
te S
core
NoneToT/FuelHazardFull
Quantitative Analysis (3)
• Only none had sig. improvement with time• Hazard better than none and time/fuel < 55 sec, sig. at 40 sec• None outperforms hazard and time/fuel at 55 sec, while is the
worst at 20 sec• Performance decreases at times dependent on auto level
InteractionEffects
optimalscore
IncreasingPerformance
13
• Idea of “Characteristic Time” (CT)– Period of beneficial
time from having auto assistance
• Possible linear metric– Similar slopes
– Intercept quantifies relative auto benefit
• CTFull > 55 sec
• CTHazard 55 sec
• CTTime/Fuel 20 sec
Temporal Benefit of Automation
y = -0.0028x + 0.4486
y = -0.0032x + 0.1783
y = -0.0026x + 0.0589
-0.15
-0.05
0.05
0.15
0.25
0.35
0.45
10 20 30 40 50 60Time (sec)
Au
to B
enef
it (
No
ne
- A
uto
)FullHazardToT & Fuel
Increasing Auto Benefit
14
Failure Analysis
• Most failures with hazard–Contrary to quantitative performance
• 40 sec had fewest failures–While no quantitative improvement in performance after 28 sec
• Failures at 55 sec 20 secFailure Count Time Automation 20 28 40 55 Grand Total
None 2 1 1 2 6ToT & Fuel 0 3 2 3 8
Hazard 6 2 0 5 13Full 2 1 1 1 5
Grand Total 10 7 4 11 32
A failed scenario had 1 of the following:1. Hit a brown hazard2. Arrived target outside of time window3. Not enough fuel to complete mission
0
1
2
3
4
5
0 1 2 3 4 5 6
Failure Count
Su
bje
ct F
req
uen
cy
• Only 1 subject was perfect• 14.3% failure rate, 32 of 224
15
Conclusions
• Automation does assist in time-critical decision making– Auto benefit decreases as available time
increases
– Auto benefit increases dependent on type and amount of information integrated by automation
• Moderate amounts of time may actually hinder performance over less time
• Subject data supported quantitative analysis
16
Future Work
• Develop a generalized model for decision support automation
• Identify information support needs of human
• Follow-on experiment– More specific to flight environment– Data points > 55 seconds to reach
characteristic time– Further test interactions between partial
and no automation