Post on 24-Dec-2015
Psychophysics 3
Research Methods
Fall 2010
Tamás Bőhm
Signal detection theory
• Aka. sensory decision theory (SDT)• A model & a data analysis method for decision
problems with uncertainty (noise)• Originates from World
War II: aircraft detection on radar signals
• Today: widely used in psychophysics, medicine, radiology and machine learning
Signal detection theory
• Experiment setup:– In some trials a stimulus (signal) is presented, in
others there is no stimulus;– Observer reports if she/he saw a signal or not– Calculate how many times the observer detected a
signal when she/he was presented one (hit rate)• Is the hit rate all we want to know?
Two observers achieved the same hit rate. Are they certainly behaving the same way?
• NO, we also need to know how many times the observer said “I see” when there was no signal (false alarm rate)
Signal detection theory
• Confusion matrix: contains all the information about the observer’s performance
Signal detection theory
• Confusion matrix: contains all the information about the observer’s performance
• As columns add up to 100%, it is enough to know one item from each column
40 trials
20 20
18
2
6
14
= 100% = 100%
= 90%
= 10% = 70%
= 30%
Signal detection theory
• Perfect detection:
100%
100%0%
0%
Signal detection theory
• No detection at all (1st example): always reporting “Seen”
100%
0%0%
100%
Signal detection theory
• No detection (2nd example): always reporting “Not seen”
0%
100%100%
0%
Signal detection theory
• No detection (3rd example): flipping a coin
50%
50%50%
50%
Signal detection theory
• No detection (4th example): reporting “Seen” in 30% of the trials (no matter what is presented)
30%
70%70%
30%=
=
Rows equal no detection
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
90% 30%
10% 70%
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
100% 0%
0% 100%
Perfect detection
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
100% 100%
0% 0%
No detection: always “yes”
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
0% 0%
100% 100%
No detection: always “no”
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
50% 50%
50% 50%
No detection: reporting “yes” in 50% of the trials (flipping a coin)
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
40% 40%
60% 60%
No detection: reporting “yes” in 40% of the trials
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
30% 30%
70% 70%
No detection: reporting “yes” in 30% of the trials
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%
60% 60%
40% 40%
No detection: reporting “yes” in 60% of the trials
Signal detection theory
• Receiver operating characteristic (ROC):
false alarm rate
hit
rat
e
100%
100%Diagonal: no detection
Signal detection theory
• SDT model:
• No way to remove the noise• But sensation can be separated from decision by
using ROCs
Sensation
Noise
DecisionSignal
present/absent
Sensation level (SL)
SL ≥ β
Criterion (β)
SL < β
YES
NO
Signal detection theory
Sensation
(Noise)
DecisionSignal
present/absent
Sensation level (SL)
SL ≥ β
Criterion (β)
SL < β
YES
NO
sensation level
pro
bab
ility
Without noise: perfect detection is possible
criterion signal present
signal absent
Signal detection theory
Sensation
(Noise)
DecisionSignal
present/absent
Sensation level (SL)
SL ≥ β
Criterion (β)
SL < β
YES
NO
sensation level
pro
bab
ility
criterion signal present
signal absent
100% 0%
0% 100%
Signal detection theory
Sensation
Noise
DecisionSignal
present/absent
Sensation level (SL)
SL ≥ β
Criterion (β)
SL < β
YES
NO
sensation level
pro
bab
ility
Noise: smears the distributions perfect detection is impossible (if the two distributions overlap)
signal absent(noise only)
signal present(signal+noise)criterion
online demo
Signal detection theory
Sensation level
Sensation level
http://www-psych.stanford.edu/~lera/psych115s/notes/signal/
Signal detection theory
Sensation level
Sensation level
false alarm rateh
it r
ate
Signal detection theory
false alarm rateh
it r
ate
ROC curve
β = 8
β = 6
β = 10
β = 6
β = 8
β = 10
Signal detection theory
false alarm rateh
it r
ate
β
sensation level
pro
bab
ility
• Criterion (β): specifies where we are on the ROC curve
• The ROC curve is specified by sensory capacities only(discriminability)
Signal detection theory
• Discriminability: how well the observer can separate the presence of signal from its absence~ overlap between the two distributions~ bowing out of the ROC curve
• Measured by d’ (discriminability index,also called sensitivity)
http://www-psych.stanford.edu/~lera/psych115s/notes/signal/
Signal detection theory
d’: selects the ROC curve
β: specifies a point on the selected ROC curve
same information as hit rate & false alarm rate, but:
hit rate, false alarm rate:both reflect sensation & decision characteristics;cannot separate the two
d’: depends only on sensation
β: depends only on decision
β
The two processes are separated
http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.ROC.html
Signal detection theory
Fechner’s methods:Is a stimulus detectable? Yes or no?
• Clear-cut threshold value (with some variability) that can be measured– Stimulus intensity >
threshold detectable– Stimulus intensity <
threshold not detectable
• Dichotic outcome, categorical model
Signal detection theory:How well is it detectable? How sensitive the observer is to the stimulus?
• Measured by d’– The higher d’ is, the more
the stimulus is detectable– d’ = 0
not detectable at all
• Scalar outcome, dimensional model
Signal detection theory
Sensation
(Noise)
Stimulus
Sensation level (SL)
Different task
Correct Incorrect
Forced-choice: eliminates the criterion
SDT: separates the criterion
Decision
SL ≥ β
Criterion (β)
SL < β
YES
NO
• Problem with Fechner’s methods: criterion
Signal detection theory
Psychophysical measurements with SDT:1. Create a stimulus set with a range of intensities (like
in the method of constant stimuli)2. Test each stimulus many times with each observer3. On each trial, either present a randomly selected
stimulus or do not present anything4. Ask the observer if he/she detected the stimulus5. Calculate the hit rate and false alarm rate for each
observer, for each stimulus intensity6. Use the formula/table to calculate d’ for each case7. Examine how d’ changes with intensity: the higher
d’ is for a stimulus intensity, the greater the observer’s ability to detect this intensity
http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.SignalDetection.html
Signal detection theory
• Main results: changes in d’ values
Caudek–Rubin Vision Res. 2001
Signal detection theory
• There is also a β value for each d’ value• It can be informative about the decision
behavior:– Balanced: false alarm and
miss rates are equal– Liberal: the observer
says “yes” whenever there may be a signal
– Conservative: decision is yes only when it is almost certain that there is a signal
sensation level
pro
bab
ility
balanced
conservativeliberal