Naoshige Uchida - US grants

DESCRIPTION (provided by applicant): The midbrain dopamine system is critical for learning, motivation and processing rewards. Malfunctions of this system are associated with a variety of pathological conditions including depression, schizophrenia and addiction. Dopamine neurons, located in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc), are thought to broadcast reward prediction error (RPE) signals, i.e., the discrepancy between actual reward and expected reward. Furthermore, recent studies have indicated that dopamine neurons in the VTA and SNc convey different signals, value and saliency, respectively. While these observations have generated great interest, how dopamine neurons compute error signals is unknown. This project will address the following two main questions: (1) How do dopamine neurons compute RPE signals? and (2) What underlies the different response properties of VTA and SNc dopamine neurons? Although local GABAergic neurons in VTA and SN exert a powerful influence on dopamine neurons, little is known about their firing patterns in a behavioral context. Specific Aim 1 will test the hypothesis that these GABAergic neurons encode reward expectation, which contributes to the prediction error calculations of dopamine neurons. To test this hypothesis, the spiking activity from VTA will be recorded while mice perform a classical conditioning paradigm in which they associate different odors with different outcomes (big water, small water, nothing and airpuff). To identify neurotransmitter types of recorded neurons, dopaminergic or GABAergic neurons will be tagged with channelrhodopsin (ChR2) and whether recorded neurons respond to light will be examined. First, whether identified dopamine neurons indeed convey RPE signals will be examined. Second, whether VTA GABAergic neurons show sustained activation reflecting upcoming reward value during the delay between a reward- predicting odor and the delivery of reward will be tested. Specific Aim 2 will test whether the reward responses of dopamine neurons under various conditions can be accounted for by the response profiles of VTA GABAergic neurons. The activity of dopaminergic and GABAergic neurons in VTA will be recorded while mice associate new odors with reward, or in a task in which the timing of reward was manipulated. Specific Aim 3 will test the hypothesis that the activity of local GABAergic neurons can explain the different response properties of dopamine neurons in the VTA and substantia nigra (SN). Specifically, the hypothesis that SN GABAergic neurons signal the prediction of aversive as well as rewarding events will be tested. In total, the results obtained in the proposed project will elucidate the key players contributing o RPE calculations of dopamine neurons. Understanding the detailed neural circuit mechanisms for RPE computation will facilitate our ability to understand etiology of depression, schizophrenia and addiction, and to design preventive and therapeutic approaches for these disorders.

DESCRIPTION (provided by applicant): Dopamine neurons in the ventral tegmental area (VTA) play central roles in learning and motivation. In tasks involving rewards, they respond in a stereotyped fashion. They are activated by unpredicted rewards. But when a sensory cue predicts reward, they instead start responding to the cue, while their response to reward attenuates. Moreover, when a predicted reward is omitted, their activity is transiently suppressed. From these observations, it has been postulated that dopamine neurons signal discrepancies between expected and actual reward, i.e., they compute the reward prediction error (RPE). However, it remains unknown how dopamine neurons compute RPE. To understand this question, it is important to know (1) what mechanisms suppress dopamine neurons' responses to reward when the reward is expected, and (2) what mechanisms are responsible for generating the response of dopamine neurons to reward-predicting cues. A previous study in our laboratory has shown that VTA GABA neurons exhibit sustained activations during the delay between a reward-predictive cue and reward. This result suggests that VTA GABA neurons suppress dopamine neurons' responses to reward when the reward is expected. In this proposal, to test this idea experimentally, the sustained activity of VTA GABA neurons will be optogenetically manipulated, and how this manipulation affects dopamine neurons' responses to reward will be examined electrophysiologically. Second, although previous studies have shown that the nucleus accumbens (NAc) and the ventral pallidum(VP) provide large numbers of inhibitory inputs to dopamine neurons, and NAc neurons project to VP, the exact roles of these connections in regulating the activity of dopamine neurons remain unclear. In a preliminary experiment, inactivation of unilateral NAc was found to greatly reduce dopamine neurons' responses to reward-predictive cues. We will experimentally test the hypothesis that a disynaptic, inhibitory pathway from NAc to VP to dopamine neurons is responsible in generating dopamine neurons' responses to reward-predictive cues. In total, this project aims to experimentally test the aforementioned specific hypotheses using integrative approaches in mice. Malfunctions of the dopamine system are associated with a variety of pathological conditions including depression, schizophrenia and addiction. By providing a detailed, circuit-level analysis of dopamine neuron firing, we will provide a framework for understanding how the brain learns from rewards, and how this system can be disrupted in disease.

Project Summary Dopamine is thought to be a key regulator of learning from appetitive as well as aversive events. It has been proposed that dopamine neurons signal value prediction error (VPE, often referred to as reward prediction error), or the difference between the values of actual and predicted outcomes. Although accumulating evidence supports this idea for reward, how dopamine neurons integrate information about aversive events remains highly controversial. Some studies have shown that aversive stimuli inhibit dopamine neurons, while others have suggested that aversive events activate at least some dopamine neurons. Others have argued that dopamine neurons largely ignore aversive events. In order to resolve the above controversy, this project aims to examine how dopamine neurons in the ventral tegmental area (VTA) convey information about aversive events . Our main hypothesis is that dopamine neurons alter the way they respond to rewarding and aversive stimuli depending on reward contexts. More specifically, we hypothesize that in low reward contexts, dopamine neurons faithfully signal value prediction errors by combining the value of both reward and aversion. In high reward contexts, we hypothesize that dopamine neurons acquire short-latency excitatory response to aversive stimuli and decrease their inhibitory responses to aversive stimuli, thereby diminishing their ability to signal value prediction errors. Aim 1 will test the specific hypothesis that dopamine neurons in VTA represent a combined value for reward and aversion along a one-dimensional value axis in a low reward context. Aim 2 will test the hypothesis that VTA dopamine neurons signal value prediction error in low but not high reward contexts. Aim 3 will test the hypothesis that expectation of an aversive stimulus reduces aversive stimulus- induced inhibition in a subtractive fashion in low reward contexts. Malfunction of the midbrain dopamine system is associated with a variety of pathological conditions including depression, anhedonia, apathy, schizophrenia, addiction, eating disorders and Parkinson's disease. In particular, aberrant dopamine responses to salient events have been implicated in addiction, schizophrenia and other mental disorders. This study will allow us to better understand situations in which dopamine neurons are activated by aversive events, and will serve as a foundation to understand dopamine signaling in health and disease.

Project abstract Dopamine plays important roles in learning and motivation. It has been thought that dopamine neurons (DANs) signal reward prediction errors (RPEs) but this idea has been challenged by some recent findings. For one, some studies found that dopamine concentration in the ventral striatum slowly ramps up on the timescale of several seconds during goal-directed navigation, and it was proposed that these slow-timescale dopamine fluctuations may encode Value instead of RPEs. Because RPEs (as defined by temporal difference error in reinforcement learning theories) are approximately the temporal derivative of Value, these proposals are incompatible with the RPE hypothesis. As discussed in Project 1, a theory predicts that ramping dopamine can be explained by RPEs in certain conditions. This project (Project 3) will put this idea into experimental tests. A set of experimental tests will be developed using virtual reality in head-fixed mice (Aim 1). Specifically, RPE and Value accounts will be dissociated by teleporting the animal to a new location associated with a different Value, or by changing the speed of the scene movement. Aim 2 will test the hypothesize that an explicit cue that indicates the proximity of reward is sufficient to induce a dopamine ramp in non-navigational contexts. Furthermore, by manipulating stimulus parameters, Aim 2 also aims at determining under what task conditions dopamine ramps or not. Aim 3 will experimentally manipulate DAN activity, and examine the function of slowly- fluctuating dopamine signals in the regulation of the activity of spiny projection neurons in the striatum and behaviors in goal-directed navigation.

Summary The ability to learn associations between a specific sensory stimulus and an outcome such as re- ward or punishment is a basic requirement for flexible behaviors. Malfunctions of this associative process may underlie various disorders such as drug addiction and binge eating. Rodents can learn novel stimulus-response associations after only a few repetitions, but the circuits that are modified during learning are largely unknown. The olfactory tubercle (OT), a part of the ventral stri- atum, is located at the interface between sensory and reward centers, receiving strong olfactory sensory input as well as dopaminergic innervation from the ventral tegmental area (VTA). It has been implicated in reward and is a recognized ?hot spot? for cocaine self-administration. These ob- servations suggest that the OT is the site of heterosynaptic plasticity to establish valence represen- tation associated with odors. The PIs have developed a behavioral paradigm in mice that allows rapid and flexible association of arbitrary odor cues with reward or aversion. Using this behavior, they have found evidence for an explicit representation of reward in the OT. In this project, the PIs will test the hypothesis that neural activity in the OT is modified during learning to reflect the va- lence of stimuli, and that dopaminergic signals from the VTA play a key role in this learning. Aim 1: To determine whether OT neurons signal explicit (odor-independent) valence signals after learn- ing. Mice will be trained to learn the arbitrarily assigned valence of a panel of odors and record spiking activity using tetrodes from the OT in behaving animals. The hypothesis tested is that there is an explicit valence representation in the activity of OT neurons and this representation emerges rapidly when novel odor associations are learned. Aim 2: To determine how reward and aversion are represented in the OT. The PIs will use aversive and rewarding stimuli to ask whether OT neu- rons represent true valence signals, or if they signal motivational salience. The hypothesis is that OT activity will be modulated in opposite directions for rewarding and aversive cues, signaling ex- plicit valence, with potential heterogeneity across OT cell types and subregions. Aim 3: To deter- mine whether dopaminergic axons targeting OT carry valence related signals that evolve during learning. The PIs will use fiber photometry and microendoscopy to record valence-related activity of dopaminergic axons in the OT and optogenetics to stimulate these axons in behaving mice. The hypothesis is that the OT receives dopamine inputs that represent value prediction errors that can shape the valence-related activity of OT neurons. The research proposed has broad relevance for neuroscience because it will shed light on how reward-predicting signals are learnt and represent- ed in the brain, which could help devise treatments in abnormal conditions such as addiction.

Project Summary Dopamine neurons (DNs) are key regulators of motivated behaviors, and defects in dopamine signaling may underlie some psychiatric disorders including addiction, depression, and schizophrenia, as well as neurological disorders such as Parkinson's. Much of the work in this area has been based on the dogma that DNs encode reward prediction errors (RPE) and that they do so in a uniform manner. However, work from several groups, including ours, indicates that DNs projecting to different targets exhibit distinct properties and serve distinct functions. In this proposal, we aim to link the diversity of DNs defined by their connectivity and activity with their functions. One recent example, drawn from our work, is that DNs projecting to the posterior `tail' of the striatum (TS) differ in many ways from DNs projecting to the ventral striatum (VS) and other regions. VS- projecting DNs, which signal `canonical' RPEs, are activated by reward and inhibited by negative events. By contrast, TS-projecting DNs are activated by novel stimuli and a subset of negative events. Here, based on our initial results, we will compare VS- and TS-projecting DNs with regard to their (1) activity, (2) function during behaviors, and (3) mechanism underlying the generation of the activities. We will test the main hypothesis that TS-projecting DNs integrate a unique set of inputs, signal threat prediction errors, and positively reinforces threat predictions or avoidance behaviors. Specific Aim 1 will characterize the activity of projection-specific DNs during behavior. Specific Aim 2 will demonstrate the causal link between the function and the activity patterns of projection-specific DNs. Specific Aim 3 will aim to understanding neural circuit mechanisms that generate distinct response patterns of DNs in a projection-specific manner. If our main hypothesis holds true, the results will demonstrate that dopamine in VS and TS operates in a similar manner at the algorithmic level: in both systems, an increase in dopamine results in facilitation of certain behaviors (approach or avoidance) or stimulus-based predictions (of outcome value or a threat). The methods and results of the proposed study will pave the way for looking at other DN populations in the future. This will further the goal of elucidating a unifying theory regarding the computational algorithm by which multiple DN populations function. We expect these results to provide insights into dopaminergic defects and even dopamine-directed therapeutic interventions in brain disorders.

Project abstract Animals, including humans, interact with their environment via self-generated and continuous actions that enable them to explore and subsequently experience the positive and negative consequences of their actions. As a result of their interactions with the environment, animals alter their future behavior, typically in a manner that maximizes positive and minimizes negative outcomes. Furthermore, how an animal interacts with its environment and the actions that it chooses depend on its current environment, its past experience in that environment, as well as its internal state. Thus, the actions taken by an animal are dynamic and evolving, as necessary for behavioral adaptation. It is thought that both the execution of actions, in particular goal-oriented actions, and the modification of future behavior in response to the outcome of actions, depend on evolutionarily old parts of the brain called the basal ganglia. Within the basal ganglia, cells that produce dopamine have a profound influence on behavior, including human behavior, and their activity appears to encode for features of the environment and animal experience that are important for directing goal-oriented behavior. Here we bring together a team of experimental and computational neurobiologists to understand how these dopamine- producing cells modulate behavior and basal ganglia circuitry. We will use unifying theories and models to integrate information acquired over many classes of behavior. Completing the proposed work, including the technical advances and biological discoveries, will provide a platform for future analyses of related circuitry and behaviors in many species, including humans.

Project Summary The field of artificial intelligence (AI) has recently made remarkable advances that resulted in new and improved algorithms and network architectures that proved efficient empirically in silico. These advances raise new questions in neurobiology: are these new algorithms used in the brain? The present study focuses on a new algorithm developed in the field of reinforcement learning (RL), called distributional RL, which outperforms other state-of-the-art RL algorithms and is regarded as a major advancement in RL. In environments in which rewards are probabilistic with respect to its occurrence and size, traditional RL algorithms have focused on learning to predict a single quantity, the average over all potential rewards. Distributional RL, by contrast, learns to predict the entire distribution over rewards (or values) by employing multiple value predictors that together encode all possible levels of future reward concurrently. Remarkably, theoretical work has shown that a class of distributional RL, called ?quantile distributional RL?, can arise out of a simple modification of traditional RL that introduces structured variability in dopamine reward prediction error (RPE) signals. This project set out to test the hypothesis that the brain utilizes distributional RL to predict future rewards. Aim 1 will explore the characteristics of distributional RL theoretically and make predictions that allow for testing distributional RL in the brain. Theoretical investigations and simulations will be used to determine how value representations in distributional RL differ from pre-existing population coding schemes for representing probability distributions (probabilistic population codes, distributed distributional codes, etc.). Aim 2 will examine the activity of neurons that are thought to signal RPEs and reward expectation and test various predictions of distributional RL. Specifically, the activity of dopamine neurons in the ventral tegmental area and neurons in the ventral striatum and orbitofrontal cortex will be compared to key predictions of distributional RL. Aim 3 will use optogenetic manipulation to causally demonstrate the relationship between RPE signals and distributional codes.