Human Robot Teams: Concepts, Constraints, and Experiments
description
Transcript of Human Robot Teams: Concepts, Constraints, and Experiments
Human Robot Human Robot Teams:Teams:
Concepts, Constraints, and Concepts, Constraints, and ExperimentsExperiments
Michael A. GoodrichMichael A. GoodrichDan R. Olsen Jr.Dan R. Olsen Jr.Brigham Young Brigham Young
UniversityUniversity
Research AgendaResearch Agenda Evaluation TechnologyEvaluation Technology
Neglect ToleranceNeglect Tolerance Behavioral EntropyBehavioral Entropy Fan-OutFan-Out
Interface DesignInterface Design Mixed Reality DisplaysMixed Reality Displays PrinciplesPrinciples HF ExperimentsHF Experiments
Autonomy DesignAutonomy Design Team-Based AutonomyTeam-Based Autonomy UAVsUAVs Perceptual LearningPerceptual Learning
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
A Special Case: A Special Case: The Robotics SpecialistThe Robotics Specialist
One soldierOne soldier Two UAVsTwo UAVs One UGVOne UGV
Can one person Can one person manage all three manage all three assets?assets?
At what level of At what level of performance?performance?
At what level of At what level of engagement?engagement?
A More General Case:A More General Case:Span of ControlSpan of Control
How many “things” can be managed by a How many “things” can be managed by a single human?single human? How many robots?How many robots?
How do we measure Span of Control in How do we measure Span of Control in HRI?HRI? Relationships between NT and ITRelationships between NT and IT
How do we compare possible team How do we compare possible team configurations?configurations? Evaluate performance-workload tradeoffsEvaluate performance-workload tradeoffs Identify performance of feasible Identify performance of feasible
configurationsconfigurations
The Most General Case: The Most General Case: Multiple Robots & Multiple Multiple Robots & Multiple
HumansHumans How many people are responsible How many people are responsible
for a single robot?for a single robot? How many robots can provide How many robots can provide
information to a single human?information to a single human?
VehicleCommander
ICV Driver
ICV
1 CL I UAV System
Platoon Headquarters Organization
PLT LDR
Medic Robotics NCO
ARV-A (L)
1 CL I UAV System
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
Neglect Tolerance:Neglect Tolerance:Neglect Time and Neglect Time and Interaction TimeInteraction Time
How long can the robot “go” without How long can the robot “go” without needing human input?needing human input?
How long does it take for a human to give How long does it take for a human to give guidance to the robot?guidance to the robot?
Neglect Time (NT)
Interaction Time (IT)
Fan-Out (Olsen 2003,2004): Fan-Out (Olsen 2003,2004):
How many homogeneous How many homogeneous robots?robots? How many interaction periods “fit” How many interaction periods “fit”
into one neglect periodinto one neglect period
Two other robots can be handled Two other robots can be handled while robot 1 is neglectedwhile robot 1 is neglected
Fan-out = 3Fan-out = 3
NT
IT IT IT
1
2 3 4
Can a human manage team Can a human manage team T T ? ?
Fan-out and FeasibilityFan-out and Feasibility Fan-out (homoeneous teams)Fan-out (homoeneous teams)
Feasibility (heterogeneous teams)Feasibility (heterogeneous teams)
These are upper boundsThese are upper bounds
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
Neglect Impact CurvesNeglect Impact Curves A task is A task is NeglectedNeglected if if attention is attention is elsewhereelsewhere
Neglect impacts Neglect impacts task performance: task performance: 22ndndary tasksary tasks
T im e-o ff-tas k
R o b o t E ffec tivenes s
Teleoperation
Point-to-Point
Autonom ous
Not Neglect Tolerant Not Neglect Tolerant EnoughEnough
Too Neglect TolerantToo Neglect Tolerant
Old Glory InsuranceOld Glory Insurance
Interface Efficiency Interface Efficiency CurvesCurves
Recovery from “zero” Recovery from “zero” pointpoint
Imprecise switch costsImprecise switch costs
T im e-o n-tas k
R o b o t E ffec tivenes s
Teleoperation Point-to-point
W aypoints
Efficient InterfacesEfficient Interfaces PDA-based UAV control (versus PDA-based UAV control (versus
command line)command line)
Efficient InterfacesEfficient Interfaces Phycon-based UAV control (versus Phycon-based UAV control (versus
command line)command line)
Finding NT and IT from the Finding NT and IT from the curvescurves
ExampleExample Vary minimum performance levelVary minimum performance level MeasureMeasure
Average performanceAverage performance Neglect timeNeglect time Interaction timeInteraction time
Validation of Method: Validation of Method: ComplexityComplexity
As complexity goes As complexity goes up, NT goes down up, NT goes down and IT goes upand IT goes up
Feasibility using Feasibility using NT/IT needs more NT/IT needs more workwork
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
Existing TradeoffsExisting Tradeoffs
Ideal
Increasing Threshold
Types of AutonomyTypes of Autonomy
Using Tradeoffs to Select a Using Tradeoffs to Select a ConfigurationConfiguration
Ideal
Ideal
Ideal
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
Predicting Performance of a Predicting Performance of a Heterogeneous TeamHeterogeneous Team
Each robot may have multiple autonomy Each robot may have multiple autonomy modes and interaction methodsmodes and interaction methods
Each interaction scheme yields NT, IT, and Each interaction scheme yields NT, IT, and average performance valuesaverage performance values
Two interaction schemes Point to point (P) Region of Interest (R)
Three robots
Experiment 23 subjects 148 trials 3 world complexities
Accuracy of Predictions Accuracy of Predictions in a Three-Robot Teamin a Three-Robot Team
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
What are switch costs?What are switch costs? The biggest unknown influence on span of The biggest unknown influence on span of
controlcontrol They come in several flavors:They come in several flavors:
Time to regain situation awarenessTime to regain situation awareness Time to prepare for switchTime to prepare for switch Errors and Change BlindnessErrors and Change Blindness
What really happens here?
Before and AfterBefore and After
Getting a Feel for the Getting a Feel for the ExperimentExperiment
Preliminary ResultsPreliminary Results
0123456789
10
Median Time to Recover
BlankToneTetrisUAV
ToneTone TetrisTetris UAVUAV
BlankBlank p<15p<15%%
p<20p<20%%
p<5%p<5%
ToneTone p>35p>35%%
p<15p<15%%
TetrisTetris p<10p<10%%
6 subjects, none naïve6 subjects, none naïve 207 correct change detections207 correct change detections One-sided T-test, equal variancesOne-sided T-test, equal variances
Important TrendsImportant Trends Differences not just from “time away”Differences not just from “time away”
blank and tetris have same timeblank and tetris have same time UAV and tone have same timeUAV and tone have same time Averages nearly identicalAverages nearly identical
Differences not just from “counting”Differences not just from “counting” UAV and tone both countUAV and tone both count
Differences not just from “motor channel” Differences not just from “motor channel” UAV and tone both selectUAV and tone both select Tetris requires interactionTetris requires interaction
Probably spatial reasoning and changing Probably spatial reasoning and changing perspectivesperspectives
The Presentation AgendaThe Presentation Agenda
The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals
How Many Robots?How Many Robots? AssumptionsAssumptions
Goal: Gather battle-related information Goal: Gather battle-related information while minimizing risk while minimizing risk
Media: Mostly camera/video Media: Mostly camera/video informationinformation
PredictionPrediction Interpreting camera information Interpreting camera information
difficultdifficult High robot autonomy won’t help enoughHigh robot autonomy won’t help enough
A Special Case: A Special Case: The Robotics SpecialistThe Robotics Specialist
Can one person Can one person manage multiple manage multiple robot assets?robot assets?
At what level of At what level of performance?performance?
Goal: gather Goal: gather informationinformation
Media: visual Media: visual (camera/video)(camera/video)
Belief: autonomy will Belief: autonomy will help, but not enoughhelp, but not enough
Mixed Reality DisplaysMixed Reality Displays
Eliminate “The world Eliminate “The world through a soda straw”through a soda straw”
Integrate vision with Integrate vision with active sensorsactive sensors
Integrate display with Integrate display with autonomyautonomy
Include sensor Include sensor uncertaintyuncertainty
Control pan-and tiltControl pan-and tilt Study time delay effectsStudy time delay effects
Real World ResultsReal World Results
ObjectiveObjective 51% Faster (p < .01)51% Faster (p < .01) 93% Less Safeguarding (p < .01)93% Less Safeguarding (p < .01) 29% Lower Entropy (p < . 05)29% Lower Entropy (p < . 05) 10% Better on Memory Task (p < .05)10% Better on Memory Task (p < .05)
SubjectiveSubjective 64% Less Workload / Effort (p < .001)64% Less Workload / Effort (p < .001) 70% More Learnable (p < .0001)70% More Learnable (p < .0001) 46% More Confident (p < .05)46% More Confident (p < .05)
Several Thousand WordsSeveral Thousand Words
Experiment ResultsExperiment Results
Mixed Reality Displays Mixed Reality Displays (Pan and Tilt)(Pan and Tilt)
Control the Information Control the Information Source, Source,
Not the RobotNot the Robot Phlashlight ConceptPhlashlight Concept What will UAV see?What will UAV see?
Semantic Maps and Change Semantic Maps and Change HighlightingHighlighting
Video in contextVideo in context Icon-based maps Icon-based maps
w/ semantic w/ semantic labelslabels
““That was then, That was then, this is now this is now comparison” --- comparison” --- change change highlightinghighlighting
Information Information decaydecay
Information in ContextInformation in Context
Prompt prospective memoryPrompt prospective memory Shift in a timely wayShift in a timely way Give time to prepareGive time to prepare
Robot Progress While User is Doing Secondary Task
0
2
4
6
8
10
12
14
Forced Paced w/oPath Planner
Forced Paced w PathPlanner
Self Paced w/o PathPlanner
Self Paced w PathPlanner
Seco
nds
doin
g Se
cond
ary
Task
Stopped
Going
“Neglect Tolerance
”
Support Timely ShiftsSupport Timely Shifts
Human Reaction to Robot Getting Stuck
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Forced Paced w/opath planner
Forced Paced w/Path Planner
Self Paced w/opath planner
Self Paced w/Path Planner
Sec
onds
for H
uman
to R
espo
nd
w/o Attention Management
w/ Attention Management
“Situation Awareness”
Supporting Task Supporting Task Switching: Etc.Switching: Etc.
History trails. History trails. Knowing recent past helpsKnowing recent past helps Tail on a map-based interfaceTail on a map-based interface Virtual descent into video-based interfaceVirtual descent into video-based interface Change highlighting/morphingChange highlighting/morphing
Plans: Plans: Knowing intention helpsKnowing intention helps Planned path on map-based interfacePlanned path on map-based interface Predicted trajectory on video-based interfacePredicted trajectory on video-based interface Quickened displaysQuickened displays
Task relationships: Task relationships: Knowing relationship Knowing relationship between two tasks helpsbetween two tasks helps Relative spatial location on map-based Relative spatial location on map-based
interfaceinterface Picture-in-picture on video-based interfacePicture-in-picture on video-based interface Progress bar of task X on task Y’s displayProgress bar of task X on task Y’s display
Improve Perception and Improve Perception and Scene Interpretation Scene Interpretation
(Olsen)(Olsen) Use interaction and machine Use interaction and machine
learning to make this robustlearning to make this robust
Future Concept Future Concept (Proposed)(Proposed)
Safe/Unsafe Safe/Unsafe occupancy gridsoccupancy grids Evolutionary image Evolutionary image
classifierclassifier Evolutionary Evolutionary
integration of vision integration of vision and lasersand lasers
Particle-based inverse Particle-based inverse perspective transformperspective transform
Path planningPath planning Uncertainty-based Uncertainty-based
triggers for retrainingtriggers for retraining
Learning interface Learning interface mappings from implicit mappings from implicit user cuesuser cues
ConclusionsConclusions
We can evaluate team feasibilityWe can evaluate team feasibility We can predict team performanceWe can predict team performance We need to understand task switching We need to understand task switching
betterbetter
We need to support realistic task We need to support realistic task switchingswitching Via interfacesVia interfaces Via autonomyVia autonomy
Near-Term Future WorkNear-Term Future Work
Complete validation of task Complete validation of task switching experiment paradigmswitching experiment paradigm
Compare “new and improved” Compare “new and improved” interfaces against baselineinterfaces against baseline
Compare effects of type and size of Compare effects of type and size of interfaceinterface
Answer the questions for the special Answer the questions for the special casecase