Page 97 - Science

RESEARCH | REPORT

necessary for motor and neuroprosthetic learning selection by consequence operates on neural ac- 29. R. W. Eaton, T. Libey, E. E. Fetz, J. Neurophysiol. 117, 1112–1125
(5, 14, 34). Actor-critic reinforcement learning tivity in the brain (45). (2017).
models (35, 36) suggest two sites for VTA- 30. R. Legenstein, S. M. Chase, A. B. Schwartz, W. Maass,
J. Neurosci. 30, 8400–8410 (2010).
modulated plasticity: the dorsal striatum (actor), REFERENCES AND NOTES 31. R. Héliot, K. Ganguly, J. Jimenez, J. M. Carmena,
which contributes to selection of actions (M1 1. E. L. Thorndike, Psychol. Rev. 2,1–107 (1898). IEEE Trans. Syst. Man Cybern. B Cybern. 40,1387–1397
2. B. F. Skinner, The Behavior of Organisms: An Experimental
neural activity patterns), and the ventral striatum Analysis (Appleton-Century, Oxford, 1938). (2010).
(critic), which may evaluate actions on the basis 3. R. G. Cohen, D. Sternad, Exp. Brain Res. 193,69–83 32. S. Yagishita et al., Science 345, 1616–1620 (2014).
of the value of the environmental states reached (2009). 33. W. Shen, M. Flajolet, P. Greengard, D. J. Surmeier, Science 321,
848–851 (2008).
(auditory feedback). Plasticity in dorsal striatum 4. L. Shmuelof, J. W. Krakauer, P. Mazzoni, J. Neurophysiol. 108, 34. F. J. Santos, R. F. Oliveira, X. Jin, R. M. Costa, eLife 4, e09423
578–594 (2012).
could be mediated by glutamatergic input from 5. R. M. Costa, D. Cohen, M. A. L. Nicolelis, Curr. Biol. 14, (2015).
contralateral M1 and dopaminergic input signal- 1124–1134 (2004). 35. Y. Takahashi, G. Schoenbaum, Y. Niv, Front. Neurosci. 2,86–99
(2008).
ing reward from the VTA (37), enabling adaptation 6. T. D. Barnes, Y. Kubota, D. Hu, D. Z. Jin, A. M. Graybiel, Nature 36. J. O’Doherty et al., Science 304, 452–454 (2004).
437, 1158–1161 (2005).
of the policy for reentering M1 activity patterns. 7. V. Y. Cao et al., Neuron 86, 1385–1392 (2015). 37. M.W.Howe,D. A.Dombeck, Nature 535,505–510
Plasticity in ventral striatum (32) could be me- 8. A. J. Peters, S. X. Chen, T. Komiyama, Nature 510, 263–267 (2016).
diated by strong bidirectional connections with (2014). 38. M. Fu,X.Yu,J.Lu,Y.Zuo, Nature 483,92–95
the VTA, enabling adaptation of the auditory 9. B. P. Ölveczky, T. M. Otchy, J. H. Goldberg, D. Aronov, M. S. Fee, (2012).
J. Neurophysiol. 106,386–397 (2011). 39. T. Xu et al., Nature 462, 915–919 (2009).
tones’ value.
10. E. E. Fetz, Science 163, 955–958 (1969). 40. A. Hayashi-Takagi et al., Nature 525, 333–338 (2015).
In addition, VTA stimulation may have in-
11. J. M. Carmena et al., PLOS Biol. 1, E42 (2003). 41. T. Gulati, L. Guo, D. S. Ramanathan, A. Bodepudi, K. Ganguly,
directly guided motor cortical plasticity. As animals 12. K. Ganguly, J. M. Carmena, PLOS Biol. 7, e1000153 Nat. Neurosci. 20, 1277–1284 (2017).
acquire motor skills and consolidate cortical ac- (2009). 42. G. Yang et al., Science 344, 1173–1178 (2014).
tivity patterns, motor memories are encoded in 13. V. R. Athalye, K. Ganguly, R. M. Costa, J. M. Carmena, Neuron 43. R. C. Williamson et al., PLOS Comput. Biol. 12, e1005141
93, 955–970.e5 (2017). (2016).
the formation of lasting dendritic spine ensem- 14. A. C. Koralek, X. Jin, J. D. Long II, R. M. Costa, J. M. Carmena, 44. B. F. Skinner, Science 213, 501–504 (1981).
bles (8, 38–40). Further, reinforcement guides the Nature 483, 331–335 (2012). 45. R. M. Costa, Curr. Opin. Neurobiol. 21, 579–586 (2011).
reactivation of neurons during sleep (41), leading 15. K. B. Clancy,A.C.Koralek, R.M.Costa,D.E. Feldman,
J. M. Carmena, Nat. Neurosci. 17,807–809 (2014).
to the formation of dendritic spines (42)aswell 16. W. Schultz, P. Dayan, P. R. Montague, Science 275, 1593–1599 ACKNOWLEDGMENTS Downloaded from
as the identification of neurons responsible for (1997). We thank A. Castro for the motor reinforcement experiments;
achieving a target pattern (41). Thus, our ob- 17. R. S. Sutton, A. G. Barto, Reinforcement Learning: An A. Koralek, P. Khanna, and R. Neely for helpful discussions;
and A. Vaz for animal colony management. This work
served changes in shared variance could also Introduction (MIT Press, 1998). was supported by a NSF Graduate Research Fellowship
reflect sleep-dependent changes in motor corti- 18. E. E. Steinberg et al., Nat. Neurosci. 16, 966–973 to V.R.A.; grants from the NSF (CBET-0954243 and
(2013).
cal synaptic connectivity. Recent modeling work 19. J. Olds, P. Milner, J. Comp. Physiol. Psychol. 47, 419–427 EFRI-M3C 1137267) and Office of Naval Research (N00014-15-
1-2312) to J.M.C.; and grants from the European Research
shows that excitation-inhibition–balanced spiking (1954). Area ERA-NET, European Research Council (COG 617142),
networks with clustered connectivity exhibited 20. D. Corbett, R. A. Wise, Brain Res. 185,1–15 (1980). and Howard Hughes Medical Institute (IEC 55007415)
prominent low-dimensional shared variance, 21. H.-C. Tsai et al., Science 324, 1080–1084 (2009). to R.M.C. Data presented in this paper can be found on
whereas nonclustered networks exhibited weak, 22. I. B. Witten et al., Neuron 72, 721–733 (2011). figshare at https://doi.org/10.6084/m9.figshare.5687101.v1. http://science.sciencemag.org/
high-dimensional shared variance (43). 23. S. Gong et al., J. Neurosci. 27, 9817–9823 (2007).
24. L. Grosenick, J. H. Marshel, K. Deisseroth, Neuron 86, 106–139
Our results provide causal evidence for a neu- (2015). SUPPLEMENTARY MATERIALS
ral law of effect, describing how the brain selects 25. A. P. Dempster, N. M. Laird, D. B. Rubin, J. R. Stat. Soc. B 39, www.sciencemag.org/content/359/6379/1024/suppl/DC1
and shapes neural activitypatternsthrough neu- 1–38 (1977). Materials and Methods
ral reinforcement. As Skinner noted, selection by 26. B. S. Everitt, An Introduction to Latent Variable Models Figs. S1 to S12
(Chapman and Hall, London, 1984).
consequence is a mechanism driving the evolu- References (46–49)
27. B. M. Yu et al., J. Neurophysiol. 102,614–635
tion of living things, from species to societies (2009). 8 August 2017; accepted 8 January 2018
to behavior (44). Our results help uncover how 28. P. T. Sadtler et al., Nature 512, 423–426 (2014). 10.1126/science.aao6058 on March 1, 2018

Athalye et al., Science 359, 1024–1029 (2018) 2 March 2018 6of 6