Samuel Gershman - US grants

Project abstract According to the standard reinforcement learning framework, the basal ganglia implements estimation of long- term future reward and the control of actions to maximize future reward. Dopamine (DA) plays a central role by providing the learning signal (reward prediction error, or RPE) that guides updating of reward predictions and the action policy. Despite its success, the reinforcement learning framework has been challenged from a number of directions. Some studies have suggested that DA encodes reward predictions themselves, rather than reward prediction errors, and other studies have suggested that DA may play a role in invigorating action selection independently from its contribution to learning. A major goal of this project is to develop a reinforcement learning theory of basal ganglia function that addresses these challenges, and more broadly presents a unifying view of how learning, probabilistic inference, and action selection work together to produce adaptive behavior. Our theoretical innovation can be divided into three components. First, we argue that cortical inputs to the striatum encode a probability distribution over hidden states, known as the belief state. Second, we argue that striatal projection neurons transform this input through a set of basis functions, whose purpose is to facilitate reward prediction. The synaptic weights that parametrize these predictions are updated based on the DA RPE signal. Third, we argue that action selection circuits in the dorsal striatum use probabilistic information about rewards to implement uncertainty-guided exploration.

Project abstract Animals, including humans, interact with their environment via self-generated and continuous actions that enable them to explore and subsequently experience the positive and negative consequences of their actions. As a result of their interactions with the environment, animals alter their future behavior, typically in a manner that maximizes positive and minimizes negative outcomes. Furthermore, how an animal interacts with its environment and the actions that it chooses depend on its current environment, its past experience in that environment, as well as its internal state. Thus, the actions taken by an animal are dynamic and evolving, as necessary for behavioral adaptation. It is thought that both the execution of actions, in particular goal-oriented actions, and the modification of future behavior in response to the outcome of actions, depend on evolutionarily old parts of the brain called the basal ganglia. Within the basal ganglia, cells that produce dopamine have a profound influence on behavior, including human behavior, and their activity appears to encode for features of the environment and animal experience that are important for directing goal-oriented behavior. Here we bring together a team of experimental and computational neurobiologists to understand how these dopamine- producing cells modulate behavior and basal ganglia circuitry. We will use unifying theories and models to integrate information acquired over many classes of behavior. Completing the proposed work, including the technical advances and biological discoveries, will provide a platform for future analyses of related circuitry and behaviors in many species, including humans.

A long-term goal of cognitive neuroscience is to understand which aspects of cognition are shared across individuals and which are unique to an individual. Studies of the latter are typically concerned with traits that are relatively stable over time, constituting what is referred to as a static phenotype. Phenotyping has proven to be a powerful approach for predicting behavior across time and tasks. For example, individual differences in the ability to delay gratification at age 4 years predict academic, verbal, and socioemotional competence in adolescence. But a major limitation to the predictability of such static approaches to phenotyping is that they do not capture within-individual variation. Static phenotypes are derived from performances on tasks measured at a specific time and context, whereas we know that cognitive performances (and brain measures of it) vary within individuals in relatively short time frames depending on such factors as sleep, stress, mood, alertness, and motivation. To predict an individual's cognitive performance across time, one needs to understand how the individual's cognitive state changes and what drives those changes. This research project, conducted by investigators at Harvard University, will fill this gap by collecting individual data repeatedly over time. By fitting computational models to the data, the researchers will extract a dynamic "computational phenotype? of each individual. They hypothesize that changes will be captured computationally by a relatively small set of dynamical parameters and that a small set of brain networks will be found to map onto those parameters. If this hypothesis is correct, then the project will have the potential to open the door to targeted, precise, and individual-specific training interventions to improve cognitive performance. This project is funded by Integrative Strategies for Understanding Neural and Cognitive Systems (NCS), a multidisciplinary program jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).

Computational phenotyping has recently emerged as a powerful technique for characterizing variation between individuals. By fitting computational cognitive models to behavioral data, investigators can use the resulting parameter estimates as a cognitive ?fingerprint? for an individual. Computational phenotypes have the advantage over other kinds of phenotypes (e.g., those based on surveys) of being more closely linked to underlying cognitive and neural mechanisms. Research has shown the utility of computational phenotyping in predicting individual-level outcomes, designing interventions, and providing an alternative to traditional diagnostic criteria. A critical limitation of this approach is that it has typically conceptualized the phenotype as a trait?a static descriptor of an individual. In the first aim, the investigators will formalize and experimentally validate a dynamic conceptualization of the computational phenotype. To accomplish this aim, the investigators will have participants complete a battery of behavioral tasks? weekly over three months ? for which established computational models exist. Data from this longitudinal study will be used to estimate how each participant?s computational phenotype uniquely changes over time, and the investigators will employ statistical methods to extract low-dimensional structure in the phenotype. In the second aim, the investigators will use longitudinal neuroimaging in conjunction with the behavioral battery to identify networks in the brain that track the low-dimensional phenotype structure, allowing them to pinpoint the neural locus of intra-individual variation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Social groups are woven tightly into the fabric of people’s lives. They shape how people perceive, punish, cooperate with, and learn from other people. This project seeks to understand how people discover the structure of social groups from patterns in the behavior of individuals. The project is centered on the concept of social structure learning. According this account, the brain uses statistical learning algorithms to sort individuals into latent groups on the basis of their behavioral patterns. These group representations are updated as more evidence is accumulated. The research extends the social structure learning model in several ways. One is to better understand the processes by which updating, subtyping, and subgrouping occur. Another is to establish how people balance the influence of explicit social categories against latent groupings. A third is to better understand how people resolve the challenge of cross-categorization. The project offers broad societal relevance by shedding light on the nature of social biases and stereotypes, ultimately pointing the way toward reducing discrimination.

This project advances basic understanding of social structure learning by using a combination of computational modeling and laboratory experiments. Computational models offer a formalization of hypotheses and make quantitative predictions about behavior. The project develops a computational model that makes specific predictions and captures several important features of social structure learning: (i) how people infer hierarchically-structured groups; (ii) how people use explicit social categories to guide their inferences about group structure; and (iii) how people infer multiple groupings of the same individuals. Integrating insights from these models into the study of social cognition allows for greater predictive precision and stimulates innovative strategies for stereotype change. The project also supports a summer internship program to involve students from diverse backgrounds, along with regular engagement in public outreach and education via print interviews, social media, blog posts, and public lectures.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.