Flexible Human-Machine Information Fusion and Perception ...
Transcript of Flexible Human-Machine Information Fusion and Perception ...
![Page 1: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/1.jpg)
1DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
J. Willard CurtisEmily DoucetteAFRL/RWWN
Zhen KanMichael McCourtSiddhartha MehtaUniversity of Florida
Chau TonNRC Research Associate
Flexible Human-Machine Information Fusion and Perception in Contested
Environments
Pablo RamirezUniversity of Texas, Austin
![Page 2: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/2.jpg)
2DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Urban Target Tracking
![Page 3: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/3.jpg)
3DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Ellipse Propagation?
![Page 4: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/4.jpg)
4DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Flexible Information Fusion: Estimation Framework Requirements
How can we combine disparate “looks” at a complex and dynamic world into a common operational picture?• Must accept widely varying
information flow rates that arrive asynchronously and out of sequence
• And provides an arbitrarily rich expression of uncertainty
• While ingesting very non-traditional (negative) perceptions
• And requires a common underlying mathematical framework that is capable of ingesting human-generated information flows
![Page 5: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/5.jpg)
5DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Beyond the Kalman Filter
Traditional approach to estimating battle state (target tracks, blue force tracks, etc.) relies on KalmanFilters• Cannot express non-Gaussian
beliefs• Can only fuse Gaussian
measurements:– No logical measurements (e.g. A target is on
the house if the lights are on)
– No negative measurements (GMTI sensor doesn’t return an hits in a region of interest)
• We proposed sample-based Bayesian filters as fundamental technology for Perception in Complex and Contested Battle-spaces…
![Page 6: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/6.jpg)
6DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Bayesian Inference in Contested Environments
Contested Environments might create false or delayed measurements…• Fast Out-of-Sequence Particle
Filtering technology– At the cost of increased memory
requirements
• Stored particles allow back-testingmeasurements for validity
– Can test whether a particular information source is sending “reliable” data
• Or elegantly removing the effect of previously fused measurements that are now known to be spurious
– Time required is only linear in the number of particles.
![Page 7: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/7.jpg)
7DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
𝑘𝑘𝑘𝑘 − 1𝑘𝑘 − 2𝑘𝑘 − 3𝑘𝑘 − 4𝑘𝑘 − 5𝑘𝑘 − 6𝑘𝑘 − 7
Time
𝑘𝑘𝑘𝑘 − 1𝑘𝑘 − 2𝑘𝑘 − 3𝑘𝑘 − 4𝑘𝑘 − 5𝑘𝑘 − 6𝑘𝑘 − 7
Time
Time𝑘𝑘𝑘𝑘 − 1𝑘𝑘 − 2𝑘𝑘 − 3𝑘𝑘 − 4𝑘𝑘 − 5𝑘𝑘 − 6𝑘𝑘 − 7
𝑘𝑘 − 5
Measurementreceived
Fast processing(weight updating only)
Full processing (filter update and predictions performed)
Two realization
s of the same
random process, equally valid.
Particle set history for a given process
Out-of-Sequence Information
![Page 8: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/8.jpg)
8DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
• Problem with Fast Measurement Processing (FDM) approach: resampling.
• If a resampling occurs at any time between 𝑘𝑘𝑚𝑚 and 𝑘𝑘, then FDM cannot work.
• Solution: keep track of the latest resampling time, 𝑘𝑘𝑟𝑟. If 𝑘𝑘𝑟𝑟 < 𝑘𝑘𝑚𝑚, then it is safe to perform the FDM. Perform the normal (slow) measurement processing otherwise.
• It can be shown that for our UGS model, the estimator obtained by using this hybrid FDM approach is consistent with a (much slower) brute-force out-of-sequence approach.
8
Out of Sequence Information
![Page 9: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/9.jpg)
9DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Bayesian Engine: Particle Filters integrated into Belief Network
Our Bayesian engine provides flexible modelling of arbitrarily complex uncertainties:• Can be compressed for
communication by marginalization over a set of kernels…
• Allows “negative information” and other unusual measurement modalities
• Allows for computing “Value of Information” via classical decision theory
• And provides hooks for human decision-aiding and risk-aware sensor management.
![Page 10: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/10.jpg)
10DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Bayesian Engine Example
Enemy Inside Perimeter?
Yes NoDetection Likelihood Map
Enemy Location
Enemy in Vehicle?
Yes No
Region Conflict State
war peacehostile
Network Compromised?
Soft Information contact reported?
EO/IR track
CommsJammed>
![Page 11: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/11.jpg)
11DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Curious Partner
• Even in all-human teams, “getting on the same page” is difficult.
• When Autonomous Systems are participating with other autonomous systems or humans its even more difficult: how can we bridge gap?
• Need method for autonomous system to do two things:
– Understand when its understanding of the situation has diverged from its teammates’
– Ask the team a relevant question to bridge the divergent world-view.
• Curious Partner consists of 3 pieces: a Bayesian Engine to model the world, a Consistency Checker algorithm to ascertain whether team members are in sync, and a Query Generator algorithm to ask a good question.
Bayesian Engine
Consistency Checker
Query Generator
![Page 12: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/12.jpg)
12DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Future Work
• How can we incorporate knowledge of the evolving network topology to provide implicit measurements to improve Bayesian Engine Situational Awareness?
• How fragile is “Curious Partner” technology to cyber threats or network degradation? How to robustify?
• Intersection of Perception/Decision Making/Cyber/Network Control: Unexplored synergies and potential fragilities!
![Page 13: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/13.jpg)
13DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Goal: Develop a human sensor model using touch interface to represent soft information in a mathematical form.
• Natural extension of human perception• Flexible to encode a large class of
information• Information encoded using single,
multiple, and directional finger strokes
Kernel Density Estimator
Touch Interface for Soft Information Modeling
![Page 14: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/14.jpg)
14DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
• Combination of single, multiple, and overlapping strokes• Flexible and natural medium – a large class of qualitatively
distinct information• Robust wrt human variability and requires no offline training
• Soft information - perceived information– Observation to perception– Socio-temporal variability in
How to obtain a measurement likelihood function from touch data?
Touch Interface for Soft Information Modeling
![Page 15: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/15.jpg)
15DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
• Perception to measurement -– Large uncertainty – longer strokes, High confidence - multiple
overlapping strokes, State gradient – orientation of strokes, +ve and –ve information, prior distribution
• Measurement likelihood function – Kernel density estimator
Point cloud to density functions
Touch Interface for Soft Information Modeling
![Page 16: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/16.jpg)
16DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Urb
an T
arge
t Tra
ckin
g
Touch Interface for Soft Information Modeling
![Page 17: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/17.jpg)
17DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Perc
. Tar
get
Trac
king
Trac
king
sco
re
RM
SE
Soft Information Fusion and Sensor Tasking for Urban Target Tracking
AIAA, GNC Conference 2012. Journal submission in
preparation.
![Page 18: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/18.jpg)
18DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Commander’s Interface
Soft Information Fusion and Sensor Tasking for Urban Target Tracking
![Page 19: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/19.jpg)
19DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
• Average Reduction of Estimate Entropy is 54%
• Average Reduction of Risk Value is 43%
Mutual Information based Risk-aware Active Sensing
Min
imal
Ris
k
Ent
ropy
IterationIteration
Target State Belief Map Hazard Map Risk Map
Accepted by Systems, Man, and Cybernetics,
2015.
![Page 20: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/20.jpg)
20DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Humans are Sensors:• Provide “Soft information”
– qualitative or categorical– Voice, text, or user-interface
derived signals
• Previous work was rigid in how human perceptions could be incorporated:
– Limited vocabulary/codebook
– Softmax models
• State of the art didn’t model human physiological issues well
– Training level– Alertness/fatigue– Stress …
Soft Information Fusion
“the target is behind the tall building on my right”
“the target appears to be stationary”
𝑦𝑦𝐻𝐻
Machine
�𝑥𝑥𝑀𝑀�𝑥𝑥𝐻𝐻
�𝑥𝑥Takes final pdf and returns an estimate
𝑦𝑦𝑀𝑀
Machine Distribution Generator
Confidence Measures
𝑝𝑝𝑋𝑋𝑀𝑀𝑝𝑝𝑋𝑋𝐻𝐻
Bayesian Fusion
𝑝𝑝𝑋𝑋
HumanDistribution Generator
![Page 21: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/21.jpg)
21DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Model Human (Sensor) Performance with BBN
Total Competence
Alertness
Training
Stress
GSR
Heart Rate
Bayesian Belief Network for Human Performance as a Soft Sensor:• Uses variety of input data:
– Heart Rate– Galvanic Skin Response– Training logs or aptitude tests– Eye Tracker– EEG
• Total Competence is probability distribution over several classes:
– Very High, High, Medium, Mediocre, Poor
– Used to modify human’s soft “reports”
– Individualizable!
![Page 22: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/22.jpg)
22DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Human Sensor with Uncertainty
“I don’t see a threat”
HumanDistribution Generator
Skill level = 1( untrained)
Heart Rate = 70 bpm (bored)
Eye-tracker = 3 sec blink intvl (tired)�𝑥𝑥𝐻𝐻
𝑝𝑝𝑋𝑋𝐻𝐻
Pro
babi
lity
of T
hrea
t
Threat No-Threat
Prior Intelligence:
“Unlikely to See Threats Today”
1.0
![Page 23: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/23.jpg)
23DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
True
Pos
itive
Rate
False Positive Rate
CSP ROC curve
Individualized Likelihoods …
![Page 24: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/24.jpg)
24DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Button Press Likelihoods
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
True
Pos
itive
Rate
False Positive Rate
Button ROC curve
![Page 25: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/25.jpg)
25DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Decision Support Interface
![Page 26: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/26.jpg)
26DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Probability that target is at (xi,yj)
Probability that blast radius contains (xi,yj) given fire at (xF,yF)
Probability of hitting target given fire at
(xF,yF)
𝑃𝑃 ℎ|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹 = �𝑖𝑖,𝑗𝑗=1
𝑀𝑀,𝑁𝑁
𝑃𝑃 ℎ|𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗 𝑃𝑃 𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹
Sum over entire grid
The center (xL,yL)of the actual blast radius will deviate from the indended
center (xF,yF) according to a random normal distribution
with SD=0.25*r
(xF,yF)
(xL,yL)
Expected Risk Calculation
![Page 27: Flexible Human-Machine Information Fusion and Perception ...](https://reader031.fdocuments.us/reader031/viewer/2022022717/621a5e1fb65b7a42ce76674d/html5/thumbnails/27.jpg)
27DISTRIBUTION A. Approved for public release, distribution unlimited. (96TW-2015-0268)
Probability of damage at (xi,yj) given blast radius
contains it
Probability that blast radius contains (xi,yj) given fire at (xF,yF)
Probability of damaging (xi,yj) given fire at (xF,yF)
𝑃𝑃 𝑑𝑑𝑖𝑖,𝑗𝑗|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹 = 1 ∗ 𝑃𝑃 𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹
𝐸𝐸 𝑅𝑅(𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹) = �𝑖𝑖,𝑗𝑗=1
𝑀𝑀,𝑁𝑁
𝑑𝑑𝑖𝑖,𝑗𝑗 ⋅ 𝑃𝑃 𝑑𝑑𝑖𝑖,𝑗𝑗|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹 −𝑟𝑟ℎ⋅ 𝑃𝑃 ℎ|𝑥𝑥𝐹𝐹 ,𝑦𝑦𝐹𝐹
Expected Loss given fire at (xF,yF)
Damage at (xi,yj)
Expected total damage
Reward for hitting target
Expected reward
Expected Risk Calculation