Slow-rising and fast-falling dopaminergic dynamics jointly adjust negative prediction error in the ventral striatum.

Publisher: Wiley-Blackwell Country of Publication: France NLM ID: 8918110 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1460-9568 (Electronic) Linking ISSN: 0953816X NLM ISO Abbreviation: Eur J Neurosci Subsets: MEDLINE

Publication: : Oxford : Wiley-Blackwell
Original Publication: Oxford, UK : Published on behalf of the European Neuroscience Association by Oxford University Press, c1989-

Dopamine*/metabolism ; Ventral Striatum*/metabolism; Animals ; Mice ; Conditioning, Operant/physiology ; Food ; Mesencephalon/metabolism ; Reward

The greater the reward expectations are, the more different the brain's physiological response will be. Although it is well-documented that better-than-expected outcomes are encoded quantitatively via midbrain dopaminergic (DA) activity, it has been less addressed experimentally whether worse-than-expected outcomes are expressed quantitatively as well. We show that larger reward expectations upon unexpected reward omissions are associated with the preceding slower rise and following larger decrease (DA dip) in the DA concentration at the ventral striatum of mice. We set up a lever press task on a fixed ratio (FR) schedule requiring five lever presses as an effort for a food reward (FR5). The mice occasionally checked the food magazine without a reward before completing the task. The percentage of this premature magazine entry (PME) increased as the number of lever presses approached five, showing rising expectations with increasing proximity to task completion, and hence greater reward expectations. Fibre photometry of extracellular DA dynamics in the ventral striatum using a fluorescent protein (genetically encoded GPCR activation-based DA sensor: GRAB DA2m ) revealed that the slow increase and fast decrease in DA levels around PMEs were correlated with the PME percentage, demonstrating a monotonic relationship between the DA dip amplitude and degree of expectations. Computational modelling of the lever press task implementing temporal difference errors and state transitions replicated the observed correlation between the PME frequency and DA dip amplitude in the FR5 task. Taken together, these findings indicate that the DA dip amplitude represents the degree of reward expectations monotonically, which may guide behavioural adjustment.
(© 2023 The Authors. European Journal of Neuroscience published by Federation of European Neuroscience Societies and John Wiley & Sons Ltd.)

Amo, R., Matias, S., Yamanaka, A., Tanaka, K. F., Uchida, N., & Watabe-Uchida, M. (2022). A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning. Nature Neuroscience, 25, 1082-1092. https://doi.org/10.1038/s41593-022-01109-2.
Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47, 129-141. https://doi.org/10.1016/j.neuron.2005.05.020.
Bayer, H. M., Lau, B., & Glimcher, P. W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology, 98, 1428-1439. https://doi.org/10.1152/jn.01140.2006.
Cachope, R., Mateo, Y., Mathur Brian, N., Irving, J., Wang, H.-L., Morales, M., Lovinger David, M., & Cheer Joseph, F. (2012). Selective activation of cholinergic interneurons enhances accumbal phasic dopamine release: Setting the tone for reward processing. Cell Reports, 2, 33-41. https://doi.org/10.1016/j.celrep.2012.05.011.
Chéramy, A., Romo, R., Godeheu, G., Baruch, P., & Glowinski, J. (1986). In vivo presynaptic control of dopamine release in the cat caudate nucleus-II. Facilitatory or inhibitory influence ofl-glutamate. Neuroscience, 19, 1081-1090. https://doi.org/10.1016/0306-4522(86)90124-7.
Collins, A. G. E., & Frank, M. J. (2014). Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychological Review, 121, 337-366. https://doi.org/10.1037/a0037015.
Covey, D. P., & Cheer, J. F. (2019). Accumbal dopamine release tracks the expectation of dopamine neuron-mediated reinforcement. Cell Reports, 27, 481-490.e483. https://doi.org/10.1016/j.celrep.2019.03.055.
Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., & Botvinick, M. (2020). A distributional code for value in dopamine-based reinforcement learning. Nature, 577, 671-675. https://doi.org/10.1038/s41586-019-1924-6.
Dreyer, J. K., Herrik, K. F., Berg, R. W., & Hounsgaard, J. D. (2010). Influence of phasic and tonic dopamine release on receptor activation. The Journal of Neuroscience, 30, 14273-14283. https://doi.org/10.1523/JNEUROSCI.1894-10.2010.
Glowinski, J., Chéramy, A., Romo, R., & Barbeito, L. (1988). Presynaptic regulation of dopaminergic transmission in the striatum. Cellular and Molecular Neurobiology, 8, 7-17. https://doi.org/10.1007/BF00712906.
Grieger, J. C., Choi, V. W., & Samulski, R. J. (2006). Production and characterization of adeno-associated viral vectors. Nature Protocols, 1, 1412-1428. https://doi.org/10.1038/nprot.2006.207.
Hamid, A. A., Pettibone, J. R., Mabrouk, O. S., Hetrick, V. L., Schmidt, R., Weele, C. M., Kennedy, R. T., Aragona, B. J., & Berke, J. D. (2016). Mesolimbic dopamine signals the value of work. Nature Neuroscience, 19, 117-126. https://doi.org/10.1038/nn.4173.
Hart, A. S., Rutledge, R. B., Glimcher, P. W., & Phillips, P. E. M. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. The Journal of Neuroscience, 34, 698-704. https://doi.org/10.1523/JNEUROSCI.2489-13.2014.
Heien, M., Khan, A. S., Ariansen, J. L., Cheer, J. F., Phillips, P. E. M., Wassum, K. M., & Wightman, M. R. (2005). Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. Proceedings of the National Academy of Sciences USA, 102, 10023-10028. https://doi.org/10.1073/pnas.0504657102.
Howe, M. W., & Dombeck, D. A. (2016). Rapid signalling in distinct dopaminergic axons during locomotion and reward. Nature, 535, 505-510. https://doi.org/10.1038/nature18942.
Iino, Y., Sawada, T., Yamaguchi, K., Tajiri, M., Ishii, S., Kasai, H., & Yagishita, S. (2020). Dopamine D2 receptors in discrimination learning and spine enlargement. Nature, 579, 555-560. https://doi.org/10.1038/s41586-020-2115-1.
Neumann, Jv., & Morgenstern, O. (1945). Theory of games and economic behavior. Journal of Philosophy, 42, 550-554. https://doi.org/10.2307/2019327.
Kahneman, D., & Tversky, A. (1979). On the interpretation of intuitive probability: A reply to Jonathan Cohen. Cognition, 7, 409-411. https://doi.org/10.1016/0010-0277(79)90024-6.
Ko, D., & Wanat, M. J. (2016). Phasic dopamine transmission reflects initiation vigor and exerted effort in an action- and region-specific manner. The Journal of Neuroscience, 36, 2202-2211. https://doi.org/10.1523/JNEUROSCI.1279-15.2016.
Mathiesen, S. N., Lock, J. L., Schoderboeck, L., Abraham, W. C., & Hughes, S. M. (2020). CNS transduction benefits of AAV-PHP.eB over AAV9 are dependent on administration route and mouse strain. Molecular Therapy - Methods & Clinical Development, 19, 447-458. https://doi.org/10.1016/j.omtm.2020.10.011.
Mohebi, A., Pettibone, J. R., Hamid, A. A., Wong, J.-M. T., Vinson, L. T., Patriarchi, T., Tian, L., Kennedy, R. T., & Berke, J. D. (2019). Dissociable dopamine dynamics for learning and motivation. Nature, 570, 65-70. https://doi.org/10.1038/s41586-019-1235-y.
Moutoussis, M., Bentall, R. P., Williams, J., & Dayan, P. (2009). A temporal difference account of avoidance learning. Network: Computation in Neural Systems, 19, 137-160. https://doi.org/10.1080/09548980802192784.
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004). Dopamine neurons can represent context-dependent prediction error. Neuron, 41, 269-280. https://doi.org/10.1016/S0896-6273(03)00869-9.
Natsubori, A., Tsutsui-Kimura, I., Nishida, H., Bouchekioua, Y., Sekiya, H., Uchigashima, M., Watanabe, M., d'Exaerde, A., Mimura, M., Takata, N., & Tanaka, K. F. (2017). Ventrolateral striatal medium spiny neurons positively regulate food-incentive, goal-directed behavior independently of D1 and D2 selectivity. The Journal of Neuroscience, 37, 2723-2733. https://doi.org/10.1523/JNEUROSCI.3377-16.2017.
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191, 507-520. https://doi.org/10.1007/s00213-006-0502-4.
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38, 329-337. https://doi.org/10.1016/S0896-6273(03)00169-7.
Paxinos, G., & Franklin, K. B. J. (2019). The Mouse Brain in Stereotaxic Coordinates. Academic Press.
Phillips, P. E. M., Stuber, G. D., Heien, M. L. A. V., Wightman, R. M., & Carelli, R. M. (2003). Subsecond dopamine release promotes cocaine seeking. Nature, 422, 614-618. https://doi.org/10.1038/nature01476.
Robinson, D. L., Venton, B. J., Heien, M. L. A. V., & Wightman, R. M. (2003). Detecting subsecond dopamine release with fast-scan cyclic voltammetry in vivo. Clinical Chemistry, 49, 1763-1773. https://doi.org/10.1373/49.10.1763.
Rodeberg, N. T., Sandberg, S. G., Johnson, J. A., Phillips, P. E. M., & Wightman, M. R. (2017). Hitchhiker's guide to voltammetry: Acute and chronic electrodes for in vivo fast-scan cyclic voltammetry. ACS Chemical Neuroscience, 8, 221-234. https://doi.org/10.1021/acschemneuro.6b00393.
Schultz, W., Dayan, P., & Montague, R. P. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599. https://doi.org/10.1126/science.275.5306.1593.
Starkweather, C., Gershman, S. J., & Uchida, N. (2018). The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty. Neuron, 98, 616-629.e616. https://doi.org/10.1016/j.neuron.2018.03.036.
Starkweather, C. K., & Uchida, N. (2021). Dopamine signals as temporal difference errors: Recent advances. Current Opinion in Neurobiology, 67, 95-105. https://doi.org/10.1016/j.conb.2020.08.014.
Sun, F., Zeng, J., Jing, M., Zhou, J., Feng, J., Owen, S. F., Luo, Y., Li, F., Wang, H., Yamaguchi, T., Yong, Z., Gao, Y., Peng, W., Wang, L., Zhang, S., Du, J., Lin, D., Xu, M., Kreitzer, A. C., … Li, Y. (2018). A genetically encoded fluorescent sensor enables rapid and specific detection of dopamine in flies, fish, and mice. Cell, 174, 481-496.e419. https://doi.org/10.1016/j.cell.2018.06.042.
Sun, F., Zhou, J., Dai, B., Qian, T., Zeng, J., Li, X., Zhuo, Y., Zhang, Y., Wang, Y., Qian, C., Tan, K., Feng, J., Dong, H., Lin, D., Cui, G., & Li, Y. (2020). Next-generation GRAB sensors for monitoring dopaminergic activity in vivo. Nature Methods, 1-11. https://www.nature.com/articles/s41592-020-00981-9.
Suri, R. E., & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Computation, 13, 841-862. https://doi.org/10.1162/089976601300014376.
Threlfell, S., Lalic, T., Platt Nicola, J., Jennings Katie, A., Deisseroth, K., & Cragg Stephanie, J. (2012). Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons. Neuron, 75, 58-64. https://doi.org/10.1016/j.neuron.2012.04.038.
Tian, J., & Uchida, N. (2015). Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors. Neuron, 87, 1304-1316. https://doi.org/10.1016/j.neuron.2015.08.028.
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science, 307, 1642-1645. https://doi.org/10.1126/science.1105370.
Tsutsui-Kimura, I., Natsubori, A., Mori, M., Kobayashi, K., Drew, M. R., d'Exaerde, A. K., Mimura, M., & Tanaka, K. F. (2017). Distinct roles of ventromedial versus ventrolateral striatal medium spiny neurons in reward-oriented behavior. Current Biology, 27, 3042-3048.e3044. https://doi.org/10.1016/j.cub.2017.08.061.
Tsutsui-Kimura, I., Takiue, H., Yoshida, K., Xu, M., Yano, R., Ohta, H., Nishida, H., Bouchekioua, Y., Okano, H., Uchigashima, M., Watanabe, M., Takata, N., Drew, M. R., Sano, H., Mimura, M., & Tanaka, K. F. (2017). Dysfunction of ventrolateral striatal dopamine receptor type 2-expressing medium spiny neurons impairs instrumental motivation. Nature Communications, 8, 14304. https://doi.org/10.1038/ncomms14304.
Yoshida, K., Drew, M. R., Mimura, M., & Tanaka, K. F. (2019). Serotonin-mediated inhibition of ventral hippocampus is required for sustained goal-directed behavior. Nature Neuroscience, 22, 770-777. https://doi.org/10.1038/s41593-019-0376-5.
Yoshida, K., Tsutsui-Kimura, I., Kono, A., Yamanaka, A., Kobayashi, K., Watanabe, M., Mimura, M., & Tanaka, K. F. (2020). Opposing ventral striatal medium spiny neuron activities shaped by striatal parvalbumin-expressing interneurons during goal-directed behaviors. Cell Reports, 31, 107829. https://doi.org/10.1016/j.celrep.2020.107829.
Yung, K. K. L., Bolam, J. P., Smith, A. D., Hersch, S. M., Ciliax, B. J., & Levey, A. I. (1995). Immunocytochemical localization of D1 and D2 dopamine receptors in the basal ganglia of the rat: Light and electron microscopy. Neuroscience, 65, 709-730. https://doi.org/10.1016/0306-4522(94)00536-E.
Zhou, F.-M., Liang, Y., & Dani, J. A. (2001). Endogenous nicotinic cholinergic activity regulates dopamine release in the striatum. Nature Neuroscience, 4, 1224-1229. https://doi.org/10.1038/nn769.

JP21dm0207069 Grant-in-Aid for Brain Mapping by Integrated Neurotechnologies for Disease Studies (Brain/MINDS); Agency for Medical Research and Development (AMED); 19J01068 JSPS KAKENHI; 19K06944 JSPS KAKENHI; 21H00212 JSPS KAKENHI

Keywords: GRAB sensor; dopamine; fibre photometry; operant conditioning; striatum

VTD58H1Z2X (Dopamine)

Date Created: 20230227 Date Completed: 20231221 Latest Revision: 20240101

20240101

10.1111/ejn.15945

36843200