Dynamic Representations in Visual Working Memory? · 2020. 6. 19. · Zhang & Luck (2008) Extended...
1
-500 0 500 1000 -3 -2 -1 0 1 -200 0 200 400 -3 -2 -1 0 1 -500 0 500 1000 -3 -2 -1 0 1 -200 0 200 400 600 800 -3 -2 -1 0 1 -500 0 500 1000 -5 -4 -3 -2 -1 0 1 -200 0 200 400 -5 -4 -3 -2 -1 0 1 -500 0 500 1000 -4 -3 -2 -1 0 1 -200 0 200 400 600 800 -5 -4 -3 -2 -1 0 1 Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Time after cue onset (ms) Time after track onset (ms) Time after retention-onset (ms) Shape Block Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Time after cue onset (ms) Time after track onset (ms) Time after discrete change (ms) Time after retention-onset (ms) Colour Block Diff Wave (μV) Diff Wave (μV) (1) (1) (2) (2) (3) (3) (4) (4) (5) (5) Time after discrete change (ms) -200 0 200 400 -3 -2 -1 0 1 -500 0 500 -3 -2 -1 0 1 -500 0 500 -3 -2 -1 0 1 -500 0 500 -3 -2 -1 0 1 Remember Shape Remember Both Remember Colour -500 0 500 1000 -3 -2 -1 0 1 -500 0 500 1000 -3 -2 -1 0 1 -200 0 200 400 -3 -2 -1 0 1 -500 0 500 1000 -2 -1 0 1 -200 0 200 400 600 800 -3 -2 -1 0 1 -200 0 200 400 600 800 -3 -2 -1 0 1 -200 0 200 400 600 800 -3 -2 -1 0 1 -200 0 200 400 -3 -2 -1 0 1 Time after cue-onset (ms) Time after cue-onset (ms) Time after cue-onset (ms) Time after track-onset (ms) Time after track-onset (ms) Time after track-onset (ms) Time after discrete change (ms) Time after discrete change (ms) Time after discrete change (ms) Time after retention-onset (ms) Time after retention-onset (ms) Time after retention-onset (ms) Diff Wave (μV) (1) (1) (1) (2) (2) (2) (3) (3) (3) (4) (4) (4) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) Diff Wave (μV) 0.5 0.6 0.7 0.8 0.9 1 500 1000 1500 2000 ERP (μV) Difference Wave (μV) Following stimulus change, VWM amends its existing representation of that stimulus to maintain correspondence. Following stimulus change, VWM drops its existing representation of that stimulus and re-encodes the changed stimulus to regain the lost correspondence. Dynamic Representations in Visual Working Memory? Ben Park 1 , Dirk B. Walther 1,3 , & Keisuke Fukuda 1,2 ‣ Contralateral Delay Activity (CDA) is an event-related potential whose amplitude tracks the active capacity of VWM. ‣ Visual Working Memory (VWM) is a theoretical mental workspace where we represent a limited amount of visual information in an active state for a current task. Updating Stimulus Display Participant / EEG L R Vogel & Machizawa (2004) “Remember Right!” Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748. Balaban, H., Drew, T., & Luria, R. (2019). Neural evidence for an object-based pointer system underlying working memory. cortex, 119, 362-372. Li, A. Y., Liang, J., Lee, A. C., & Barense, M. D. (2019). Visual interference can help and hinder memory: Measuring memory fidelity using a novel circular shape space. bioRxiv, 535922. Time after memory onset (ms) L R - CDA = L R Circular shape space. Li et al. (2019) Circular colour space. Zhang & Luck (2008) Extended thanks to Keisuke Fukuda, Dirk B. Walther, Fukuda Lab, BW Lab, and Rosa Torres, April Pereira, Amanda Leonetti, Sabah Rasheed, Ismael Kanca Kaan. 1 University of Toronto, 2 University of Toronto Mississauga, 3 Samsung Artificial Intelligence Center Toronto Balaban, Drew, & Luria (2018) Exp 1 - Is a loss of stimulus-to-representation correspondence sufficient for VWM representations to reset? Exp 2 - Is resetting caused by a lost correspondence in object-based representations? Discrete vs. No-Discrete Change During Track Cue Post-cue Track RI Report 500ms 500ms 1251-2000ms 1000ms Adjust Until Report Trial End Shape Block Colour Block Trial Begin No Discrete Change (Baseline) Discrete Change “Attend magenta/ cyan side” “Track the stimulus as it changes” “Report the last shape/colour you saw” How do we keep track of visual information as it changes across time? Resetting Discerning a loss of stimulus-to-representation correspondence using 2 types of change Continuous Change: ‣ Gradual change of a stimulus along a circular stimulus space (correspondence maintained). Discrete Change: ‣ Non-gradual change of a stimulus along a circular stimulus space (correspondence lost). ‣ If VWM representations reset to a loss in stimulus-to-representation correspondence, then we should observe a drop in CDA amplitude [ ] to a discrete change of a stimulus. ‣ But previous demonstrations (polygon separation) of loss of stimulus-to-representation correspondence are confounded by change in # of stimuli & change in movement direction. ‣ If VWM resetting reflects a lost correspondence in object-based representations, then we should observe a drop in CDA amplitude [ ] to a discrete change in a feature of a stimulus, regardless of its task-relevance. Discrete vs. No-Discrete Change During Track Time Cue Post-cue Track RI Report 500ms 500ms 1251-2000ms 750ms Test Until Report Remember Shape Remember Colour Remember Both Trial Begin Discrete Colour Change Discrete Shape Change Or “Report the last shape you saw” “Report the last colour you saw” Trial End “Attend left/ right side” “Track the stimulus as it changes” No Discrete Change (Control) Exp 1 Discussion References ‣ A loss of stimulus-to-representation correspondence alone is sufficient for VWM representations to reset. ‣ Maintained correspondence results in no resetting. ‣ Overall, whether VWM updates or resets depends on stimulus-to-representation correspondence. ‣ VWM resets following a discrete change in shape, irrespective of its task-relevance. ‣ Although not significant, a similar trend is observed following a discrete change in colour. ‣ Overall, the CDA's sensitivity to task-irrelevant change in a stimulus suggests that the CDA codes for object-based representations. How are VWM representations modified? -180 -90 0 90 180 0 0.005 0.01 0.015 0.02 0.025 0.03 Response proportion Response offset (°) (6) -180 -90 0 90 180 0 0.005 0.01 0.015 0.02 0.025 0.03 Response offset (°) Response proportion (6) No Discrete Change Trials Discrete Shape Change Trials Discrete Colour Change Trials No Discrete Change Trials Discrete Change Trials (1) Sustained negative amplitude in the difference wave following cue suggests participants are representing the cued stimulus in VWM. (2) Sustained negativity is amplified following track onset, likely due to stimulus motion onset (Drew & Vogel, 2014). (3) CDA amplitude drops in response to a discrete change of the stimulus, supporting the hypothesis. (4) The CDA then recovers its amplitude. (5) Following retention onset, CDA amplitude diminishes, but remains present suggesting that a VWM representation persists. (6) As foreshadowed by the full recovery of the CDA after reset, behavioural performance does not differ between discrete and no- discrete change trials. (1) Unexpectedly, a slightly positive difference wave follows cue onset. This may be due to differences in cueing stimulus. (2) Sustained negativity is observed following track onset, suggesting participants are representing the cued stimulus in VWM. (3) CDA drops to a discrete change in shape, regardless of its task-relevance. Although not significant, colour shows a similar trend. (4) Again, following retention onset, CDA amplitude diminishes, but remains present suggesting that a VWM representation persists. (5) Behavioural performance is lower when both features are task-relevant, suggesting that the number of task-relevant features influences VWM performance. Exp 2 Discussion Zhang, W., & Luck, S. J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233. Drew, T., & Vogel, E. K. (2008). Neural measures of individual differences in selecting and tracking multiple moving objects. Journal of Neuroscience, 28(16), 4183-4191. Updating = steady CDA amplitude Time Resetting = drop in CDA amplitude [ ] Does VWM recover after a reset? Accuracy Shape Test Colour Test (5) Response Time (ms) (5) Shape Test Colour Test Remember Either Remember Both Does feature-load influence VWM performance? [email protected] Baseline Baseline Correspondence lost Correspondence maintained [email protected]
Transcript of Dynamic Representations in Visual Working Memory? · 2020. 6. 19. · Zhang & Luck (2008) Extended...
-
-500 0 500 1000
-3
-2
-1
0
1-200 0 200 400
-3
-2
-1
0
1-500 0 500 1000
-3
-2
-1
0
1-200 0 200 400 600 800
-3
-2
-1
0
1
-500 0 500 1000
-5
-4
-3
-2
-1
0
1-200 0 200 400
-5
-4
-3
-2
-1
0
1-500 0 500 1000
-5
-4
-3
-2
-1
0
1-200 0 200 400 600 800
-5
-4
-3
-2
-1
0
1
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Time after cue onset (ms) Time after track onset (ms) Time after retention-onset (ms)
Shap
e Bl
ock
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Time after cue onset (ms) Time after track onset (ms) Time after discrete change (ms) Time after retention-onset (ms)
Colo
ur B
lock
Diff
Wav
e (µ
V)D
iff W
ave
(µV)
(1)
(1)
(2)
(2)
(3)
(3)
(4)
(4)
(5)
(5)
Time after discrete change (ms)
-200 0 200 400
-3
-2
-1
0
1
-500 0 500
-3
-2
-1
0
1
-500 0 500
-3
-2
-1
0
1
-500 0 500
-3
-2
-1
0
1Rem
embe
r Sh
ape
Rem
embe
r Bot
hRe
mem
ber
Colo
ur
-500 0 500 1000
-3
-2
-1
0
1
-500 0 500 1000
-3
-2
-1
0
1
-200 0 200 400
-3
-2
-1
0
1
-500 0 500 1000
-3
-2
-1
0
1
-200 0 200 400 600 800
-3
-2
-1
0
1
-200 0 200 400 600 800
-3
-2
-1
0
1
-200 0 200 400 600 800
-3
-2
-1
0
1
-200 0 200 400
-3
-2
-1
0
1
Time after cue-onset (ms)
Time after cue-onset (ms)
Time after cue-onset (ms)
Time after track-onset (ms)
Time after track-onset (ms)
Time after track-onset (ms)
Time after discrete change (ms)
Time after discrete change (ms)
Time after discrete change (ms)
Time after retention-onset (ms)
Time after retention-onset (ms)
Time after retention-onset (ms)
Diff
Wav
e (µ
V)
(1)
(1)
(1)
(2)
(2)
(2)
(3)
(3)
(3)
(4)
(4)
(4)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
Diff
Wav
e (µ
V)
1 20.5
0.6
0.7
0.8
0.9
1
1 2500
1000
1500
2000
ERP
(µV)
Diffe
renc
e Wav
e (µ
V)
Following stimulus change, VWM amends its existing representation of that stimulus to maintain correspondence.
Following stimulus change, VWM drops its existing representation of that stimulus and re-encodes the changed stimulus to regain the lost correspondence.
Dynamic Representations in Visual Working Memory?Ben Park1, Dirk B. Walther1,3, & Keisuke Fukuda1,2
‣ Contralateral Delay Activity (CDA) is an event-related potential whose amplitude tracks the active capacity of VWM.
‣ Visual Working Memory (VWM) is a theoretical mental workspace where we represent a limited amount of visual information in an active state for a current task.
Updating
Stim
ulus D
isplay
Part
icipa
nt /
EEG
L R
Voge
l & M
achi
zaw
a (2
004)
“Remember Right!”
Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748. Balaban, H., Drew, T., & Luria, R. (2019). Neural evidence for an object-based pointer system underlying working memory. cortex, 119, 362-372. Li, A. Y., Liang, J., Lee, A. C., & Barense, M. D. (2019). Visual interference can help and hinder memory: Measuring memory fidelity using a novel circular shape space. bioRxiv, 535922.
Time after memory onset (ms)
L R-CDA =
L
R
Circular shape space.Li et al. (2019)
Circular colour space. Zhang & Luck (2008) Extended thanks to Keisuke Fukuda, Dirk B. Walther, Fukuda Lab, BW Lab, andRosa Torres, April Pereira, Amanda Leonetti, Sabah Rasheed, Ismael Kanca Kaan.
1University of Toronto, 2University of Toronto Mississauga, 3Samsung Artificial Intelligence Center Toronto
Bala
ban,
Dre
w, &
Lur
ia (
2018
)
Exp 1 - Is a loss of stimulus-to-representation correspondence sufficient for VWM representations to reset? Exp 2 - Is resetting caused by a lost correspondence in object-based representations?
Discrete vs. No-Discrete ChangeDuring Track
Cue Post-cue Track RI Report500ms 500ms 1251-2000ms 1000ms
AdjustUntil Report
Trial
End
Shape Block
Colour Block
Trial
Beg
in
No Discrete Change (Baseline)
Discrete Change
“Attend magenta/cyan side”
“Track the stimulus as it changes”
“Report the last shape/colour you saw”
How do we keep track of visual information as it changes across time?
Resetting
Discerning a loss of stimulus-to-representation correspondence using 2 types of change
Continuous Change: ‣ Gradual change of a stimulus along a
circular stimulus space (correspondence maintained).
Discrete Change: ‣ Non-gradual change of a stimulus along
a circular stimulus space (correspondence lost).
‣ If VWM representations reset to a loss in stimulus-to-representation correspondence, then we should observe a drop in CDA amplitude [ ] to a discrete change of a stimulus.
‣ But previous demonstrations (polygon separation) of loss of stimulus-to-representation correspondence are confounded by change in # of stimuli & change in movement direction.
‣ If VWM resetting reflects a lost correspondence in object-based representations, then we should observe a drop in CDA amplitude [ ] to a discrete change in a feature of a stimulus, regardless of its task-relevance.
Discrete vs. No-Discrete ChangeDuring TrackTime
Cue Post-cue Track RI Report500ms 500ms 1251-2000ms 750ms
TestUntil Report
Remember Shape
Remember Colour
Remember Both
Trial
Beg
in
Discrete Colour Change
Discrete Shape Change
Or
“Report the last shape you saw”
“Report the last colour you saw”
Trial
End
“Attend left/right side”
“Track the stimulus as it changes”
No Discrete Change (Control)
Exp 1 Discussion
References
‣ A loss of stimulus-to-representation correspondence alone is sufficient for VWM representations to reset.‣ Maintained correspondence results in no resetting.‣ Overall, whether VWM updates or resets depends on stimulus-to-representation correspondence.
‣ VWM resets following a discrete change in shape, irrespective of its task-relevance. ‣ Although not significant, a similar trend is observed following a discrete change in colour.‣ Overall, the CDA's sensitivity to task-irrelevant change in a stimulus suggests that the CDA codes for
object-based representations.
How are VWM representations modified?
-180 -90 0 90 180
0
0.005
0.01
0.015
0.02
0.025
0.03
Resp
onse
pro
port
ion
Response offset (°)
(6)
-180 -90 0 90 180
0
0.005
0.01
0.015
0.02
0.025
0.03
Response offset (°)
Resp
onse
pro
port
ion
(6)
No Discrete Change Trials Discrete Shape Change Trials Discrete Colour Change Trials
No Discrete Change Trials Discrete Change Trials
(1) Sustained negative amplitude in the difference wave following cue suggests participants are representing the cued stimulus in VWM.
(2) Sustained negativity is amplified following track onset, likely due to stimulus motion onset (Drew & Vogel, 2014).
(3) CDA amplitude drops in response to a discrete change of the stimulus, supporting the hypothesis.(4) The CDA then recovers its amplitude.
(5) Following retention onset, CDA amplitude diminishes, but remains present suggesting that a VWM representation persists.
(6) As foreshadowed by the full recovery of the CDA after reset, behavioural performance does not differ between discrete and no-discrete change trials.
(1) Unexpectedly, a slightly positive difference wave follows cue onset. This may be due to differences in cueing stimulus.
(2) Sustained negativity is observed following track onset, suggesting participants are representing the cued stimulus in VWM.
(3) CDA drops to a discrete change in shape, regardless of its task-relevance. Although not significant, colour shows a similar trend.
(4) Again, following retention onset, CDA amplitude diminishes, but remains present suggesting that a VWM representation persists.
(5) Behavioural performance is lower when both features are task-relevant, suggesting that the number of task-relevant features influences VWM performance.
Exp 2 Discussion
Zhang, W., & Luck, S. J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233.Drew, T., & Vogel, E. K. (2008). Neural measures of individual differences in selecting and tracking multiple moving objects. Journal of Neuroscience, 28(16), 4183-4191.
Updating = steady CDA amplitude
Time
Resetting = drop in CDA amplitude [ ]
Does VWM recover after a reset?
Accu
racy
Shape Test Colour Test
(5)
Resp
onse
Tim
e (m
s) (5)
Shape Test Colour Test
Remember Either Remember Both
Does feature-load influence VWM performance?
BaselineBaseline
Correspondence lostCorrespondence maintained
mailto:[email protected]:[email protected]
