Objective Evaluation of Subjective Decisions

Mel Siegel & Huadong WuRobotics Institute – School of Computer Science

Carnegie Mellon University - Pittsburgh PA 15232 USA

SCIMA-2003Soft Computing Techniques in Instrumentation,

Measurement and Related Applications

Brigham Young University – Provo UT USA2003 May 17

outline• background: problem of “sensor fusion for

context aware computing”• approach: development of an “adaptive

weighted Dempster-Shafer (D-S)” algorithm• issue (= the talk’s title): objective evaluation

of subjective decisions– meta-issue: is it really an issue?

• discussion: “receiver operating characteristic”• closing the loop: ROC D-S ?

background

• “context detection” for HCI– e.g., your cell phone could ring louder if it could

know it is in your briefcase

• context detection requires subjective evaluation of “ordinary” sensor signals

• sensor fusion required when we have multiple detectors, none of them very good

• sequence of algorithms culminates in an “adaptively weighted Dempster-Shafer” method

Focus-of-AttentionFocus-of-Attention decisiondecisionby fusion of video and audio databy fusion of video and audio data

Focus-of-AttentionFocus-of-Attention decisiondecisionby fusion of video and audio databy fusion of video and audio data

Camera View

sensor fusion alternativessensor fusion alternatives

#1. complementary

#2. competitive

#3. cooperativeParametric template,Figures of merit,Syntactic pattern recognition… …

Logical templateAI rule-based reasoning,Heuristic inferenceNeural network… …

Classic InferenceSensor i: Pi( x detected | x appeared )

Simple & effective for “x vs. ¬x” problems

Priori knowledge and pdf are required to combine multiple sensor outputs, priori

assessments are not used, do not have enough reasoning power

Voting FusionAssociate pdf with confidence estimation, and

provide a way to predict the result probabilities of their boolean combinations

Though big improvement over Classic Inference method, still not powerful enough to reason at

fine granularity

Bayesian NetworkLikelihood of a hypothesis is updated using a previous likelihood estimation and additional

evidence

“cannot distinguish between lack of belief and disbelief”, cannot address a problem like “its

likely either user A or user B”

Fuzzy Logic No pdf required, very cheap in computation

It doesn’t make sense that a person is assigned as “0.6 membership of user A”, “0.7

membership of user B”, and “0.9 membership of either user A or B”

Neural NetworkFlexible, powerful, no pdf needed, cheap

computational cost in classification process

Local minimal problem, results cannot be easily explained, not suitable for dynamic

configuration of sensors

our problem: Bayes can’t do itour problem: Bayes can’t do it

head pan

left

observed pan

sensor noise

straight

left straight right

right

rightstraight

right

approach:approach:the Dempster-Shafer methodthe Dempster-Shafer method

• “Frame of Discernment” Θ lists all possibilities: {A}={ {L}, {S}, {R}, {L | S}, {S | R}, {L | R}, {L | S | R} }

•

•

,)()(Belief

AE

kii

k

EmA

AE

kiii

k

EmAA )(1)(Belief1)(tyPlausibili

)(yprobabilit:)( kki AEm

a theory of evidencea theory of evidencea theory of evidencea theory of evidenceallows belief and plausibilityallows belief and plausibilityallows belief and plausibilityallows belief and plausibility

quantifies both knowledge and ignorancequantifies both knowledge and ignorancequantifies both knowledge and ignorancequantifies both knowledge and ignorance

a generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference network

sensor fusion using “classical” sensor fusion using “classical” Dempster-Shafer Theory of Evidence Dempster-Shafer Theory of Evidence

sensor fusion using “classical” sensor fusion using “classical” Dempster-Shafer Theory of Evidence Dempster-Shafer Theory of Evidence

'

'

'

'

)()(1

)()(

)(kEkE

kjki

AkEkEkjki

EmEm

EmEm

ji Amm

{L}=0.3 {R}=0.6 {L|R}=0.1

{L}=0.4 {L}=0.4x0.3 {Φ}=0.4x0.6 {L}=0.4x0.1

{R}=0.5 {Φ}=0.5x0.3 {R}=0.5x0.6 {R}=0.5x0.6

{L|R}=0.1 {L}=0.1x0.3 {R}=0.1x0.6 {L|R}=0.1x0.1

'

'

'

'

)()(1

)()(

)(kEkE

kjki

AkEkEkjki

EmEm

EmEm

ji Amm

extension of Dempster-Shafer: extension of Dempster-Shafer: evidence weighted by sensors’ reliabilitiesevidence weighted by sensors’ reliabilities

kEjEkji

AkEjEkji

EmEm

EmEm

i

iiii

iii

Amm

mwwm

AAmwAm

)]()('[1

)]()('[

)(

)(1'

,)()('

further extension of Dempster-Shafer: further extension of Dempster-Shafer: weights change according to performance historyweights change according to performance history

overcomes sensor drift problem! overcomes sensor drift problem! overcomes sensor drift problem! overcomes sensor drift problem!

kEjEkji

AkEjEkji

EmEm

EmEm

i

iiii

iii

Amm

mtwtwm

AAmtwAm

)]()('[1

)]()('[

)(

)()()(1'

,)()()('

1.0) (0.0, factor remnance

predictionincorrect 1

predictioncorrect 0)(

)()1()(0

p

tc

ptntcptwn

ni

an arbitrary effectiveness measurean arbitrary effectiveness measure

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

focus-of-attention: meeting experiments - user subject

perc

en

tag

e r

eg

ain

ed

fro

m t

he v

isu

al sen

so

rs' lo

ss

linear probability combination standard Dempster-Shafer

Weighted Dempspter-Shafer Dynamically weighted Dempster-Shafer

generalizing via a simulation ...generalizing via a simulation ...

head pan

left

observed pan

sensor noise

straight

left straight right

right

rightstraight

right

... yields an intriguing result... yields an intriguing resultwhen sensor precisions are very differentwhen sensor precisions are very different

-10%

-5%

0%

5%

10%

15%

20%

25%

1 2 3 4 5 6

experiments

se

ns

or

fus

ion

's l

os

s/g

ain

fo

r S

1 m

iss

ed

da

ta

linear probability combination

standard Dempster-Shafer

weighted Dempster-Shafer

dynamically weighted Dempster-Shafer

the issue ...

• objective evaluation of subjective decisions– a meta-issue: is it really an issue?

“objective” vs. (?) “subjective”

• in medicine the distinction is sharp:– subjective: means what the patient tells the

physician about his/her complaint, what he/she thinks is the problem, etc

– objective: means what the physician observes (and his/her instruments report) about the condition of the patient

• statisticians talk about “rational gambling”• but in most contexts it feels fuzzier ...

• and even physicians make subjective decisions• whose quality we can evaluate objectively!:

patientreally has SARS

patient really doesn’t have SARS

physician says patient has

SARS

TRUEPOSITIVE

FALSEPOSITIVE

physician says patient doesn’t

have SARS

FALSENEGATIVE

TRUENEGATIVE

receiver operating characteristic• originally developed for target analysis• considers ratio of signal to signal-plus-

noise vs. the discriminator level set• adopted and extensively developed in the

medical diagnostic test community– { TP, TN } signal, { FP, FN } noise

– most physicians understand a test’s sensitivity == TP/(TP+FN) andspecificity == TN/(TN+FP)vs. the chosen “cut point” of the test

(dotted) ideal(dashed) useless(a) reliable(b) typical

ROC

-- increasing cut point increases TPs (good) and FNs (bad)-- decreasing cut point increases TNs (good) and FPs (bad)

closing the loop? ...

• ROC D-S ?

0 1

evidence that supports X-- fever-- white tongue-- headache

evidence that rules out X-- no virus detected-- had disease once before-- over age 55

“belief”

“plausibility”

Dempster-Shafer0 1

cut point

TP TN

FP FN

ROC

conclusions / questions• adaptive weighted D-S seems to contribute

an incremental but real improvement in appropriate sensor fusion applications

• “objective”/“subjective” distinction is fuzzy• maybe ROC and related “cut point analysis”

techniques can help us set neural net, fuzzy system, etc, parameters that are now set either arbitrarily or iteratively (hence slowly)

• is the apparent connection between D-S and ROC superficial, or real at some deep level?