Dopamine enhances model-based over model-free choice behavior
description
Transcript of Dopamine enhances model-based over model-free choice behavior
![Page 1: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/1.jpg)
Dopamine enhances model-based over model-free choice behavior
Peter Smittenaar*, Klaus Wunderlich*, Ray Dolan
![Page 2: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/2.jpg)
Model-based and model-free systems
model-free (habitual)- Cached values: single stored value- Learned over many repetitions- TD prediction error- Inflexible, but computationally cheap
model-based (goal-directed)- Model of environment with states
and rewards- Forward model computes best
action ‘on-the-fly’- Flexible, but computationally costly
Behavior is a combination of these two systems (Daw et al., 2011)
![Page 3: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/3.jpg)
How do these two systems interact to generate behavior?
Compete at output / collaborate during learning? (Daw et al., 2005; Doll et al., 2009; Biele et al., 2011)
Both systems use overlapping neural systems. (Daw et al., 2011; Wunderlich et al., 2012)
What is the role of dopamine in model-based/model-free interactions?
How does L-DOPA affect control exerted by either system?
Two systems interact
Daw et al., 2011
conjunction: model-based & model-free
![Page 4: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/4.jpg)
2-step task
based on Daw et al., 2011
X
![Page 5: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/5.jpg)
p(stay) dissociates two systems
![Page 6: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/6.jpg)
Daw et al., 2011
choices show both systems have control
![Page 7: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/7.jpg)
Daw et al., 2011
choices show both systems have control
![Page 8: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/8.jpg)
Choice is a mix of model-freeand model-based control
Daw et al., 2011
choices show both systems have control
18 subjects on and off L-DOPAwithin-subject design
![Page 9: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/9.jpg)
Daw et al., 2011
choices show both systems have control
Choice is a mix of model-freeand model-based control
![Page 10: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/10.jpg)
L-DOPA enhances model-based control
L-DOPA increases model-based, but not model-free behavior
![Page 11: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/11.jpg)
Parameter w weights MB and MF influence
Hybrid
Model-free
Model-based
V1: value stimulus 1w: weighting parameterα: model-free learning rateλ: eligibility gainr: reward on trial t
1 2
![Page 12: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/12.jpg)
L-DOPA increases model-based control (w)
* p = .005
![Page 13: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/13.jpg)
L-DOPA does not affect model-free system
L-DOPA enhances model-based over model-free control
No effect on model-free:X learning rateX noiseX policy / value updatingX positive / negative prediction errors
![Page 14: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/14.jpg)
Conclusion
L-DOPA enhances model-based over model-free control
No effect on model-free:X learning rateX noiseX policy / value updatingX positive / negative prediction errors
L-DOPA mighto improve components of model-based system o directly alter interaction between both systems at learning or choice (Doll et al., 2009)
![Page 15: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/15.jpg)
L-DOPA minus placebo
Effect stronger after unrewarded trials
![Page 16: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/16.jpg)
L-DOPA minus placebo
Effect stronger after unrewarded trials
Increase in model-based control particularly strong after unrewarded trials
![Page 17: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/17.jpg)
Conclusion
L-DOPA enhances model-based over model-free behavior
L-DOPA mighto improve components of model-based systemo directly alter interaction between both systems at learning or choice (Doll et al., 2009)
o facilitate switching to model-based control when needed (Isoda and Hikosaka, 2011)
![Page 18: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/18.jpg)
Acknowledgements
Klaus Wunderlich
Tamara Shiner
The Einstein meeting’s organizers
Ray Dolan
![Page 19: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/19.jpg)
Thank you
![Page 20: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/20.jpg)
![Page 21: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/21.jpg)
‘Random effects’ Bayesian model comparison
Alternative models
Alternative model toa, b, p, w
Placebo L-DOPABetter in #subjects
Exceedance probability
Better in #subjects
Exceedance probability
a1, a2, b1, b2, l, p, w 17 >0.999 15 0.999a1, a2, b1, b2, p, w 14 0.997 15 0.999a, b1, b2, p, w 13 0.970 14 0.998a1, a2, b, p, w 15 >0.999 15 >0.999a+, a-, b, p, w 12 >.831 15 >.996a, b, l, p, w 16 >0.999 17 >0.999a, b, w 16 >0.999 12 0.944a, b 16 >0.999 12 0.911 MF/MB learning rates 16 0.999 14 0.999Actor/critic learning 18 >0.999 17 >0.999MB prediction errors 12 >0.999 13 0.998
![Page 22: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/22.jpg)
![Page 23: Dopamine enhances model-based over model-free choice behavior](https://reader036.fdocuments.us/reader036/viewer/2022070423/5681668f550346895dda6421/html5/thumbnails/23.jpg)