Daeyeol Lee - US grants

DESCRIPTION (provided by applicant): Previous lesion and imaging studies have identified a number of brain areas involved in retrieval and production of movement sequences. However, the neural mechanisms responsible for selecting optimal movement sequences are not well understood. According to reinforcement algorithms, the outcome of a movement is evaluated with a value function, which is defined as the expected sum of temporally discounted future rewards resulting from that movement. It is also assumed that at each time step, the animal selects the movement with the maximal value function. This framework has successfully accounted for various forms of reward-related activities in multiple brain areas. Nevertheless, the possibility that reinforcement learning algorithms can account for the neural processes of sequence selection has not been previously explored. The experiments proposed in this application will first test whether the value functions of individual movements are represented in the spike rates of neurons in the frontal cortex. These experiments will be carried out in monkeys performing specific behavioral tasks, and single-unit and local field potential activities will be recorded from the lateral and medial frontal cortex using a multi-electrode recording system. Following experiments will then test the hypothesis that neurons encoding value functions of individual movements contribute to sequence selection by favoring a particular movement with a large action value, and determine whether changes in their activity during learning follow the predictions of reinforcement learning algorithms. The causal relationship between the action-value related activities and sequence selection will be also tested using the method of reversible inactivation. It is also hypothesized that the neural representation of value functions is independent of specific effectors, and this hypothesis will be tested by comparing the neural activity associated with the same eye and hand movement sequences. Finally, the hypothesis that synchronous spikes and/or phase-locked oscillation in neural activity plays a special role in transmitting signals related to movement sequences will be tested. The results from these experiments will provide a novel insight into the brain mechanisms underlying the organization of sequential behavior

DESCRIPTION (provided by applicant): Signals related to sensory stimuli and behavioral responses are distributed in the activity of a large number of neurons in multiple brain areas. In addition, the flow of information across different cortical areas must be controlled according to the behavioral context. Therefore, understanding the nature of communication among different groups of neurons and the physical basis for regulating such information flow would be an important step to optimize the diagnosis and treatment of various neurological conditions. The cortical network linking prefrontal cortex and posterior parietal cortex in primate brains provides a model system for investigating the nature of cortical communication, because the functional properties of individual neurons in these two cortical areas have been extensively studied. However, how the exchange of information across these two cortical areas is controlled by the behavioral task has not been systematically investigated. The proposed experiments will test the hypothesis that spike synchrony and coherent oscillation of neural activity plays an important role in context-dependent long-range cortical communication. Specifically, whether spike synchrony and coherent oscillation is related to the storage of spatial information in the working memory and whether such temporally related activity reflects the anticipation of behaviorally relevant sensory events will be tested. Also tested will be whether the frequency characteristics of such inter-areal interaction differ from those of local interaction among neurons in the same cortical area. Further experiments will determine whether the types of information stored in working memory influence the pattern of spike synchrony and coherent oscillation across the prefrontal and posterior parietal cortex. Finally, the proposed experiments will test whether temporally correlated activity underlies the integration of the animal's previous experience during decision-making process. These experiments will be performed in monkeys using two separate multi-electrode recording systems, and dynamic interaction among individual neurons will be analyzed using new analytical methods based on wavelet analysis and other standard statistical methods. The outcome of this research will advance our understanding on how the flow of behaviorally relevant information is regulated across a broad network of cortical neurons.

DESCRIPTION (provided by applicant): Impaired abilities to make flexible decisions characterize many mental disorders, including depression, obsessive-compulsive disorders, autism, and schizophrenia. Nevertheless, the neural mechanisms responsible for rational decision-making are poorly understood. By combining methods of computational modeling and single-neuron recording from behaving primates, this collaborative proposal seeks to obtain novel insights as to how the brain evaluates the expected outcomes of alternative actions and make optimal choices in the face of a highly dynamic and interactive environment. Sensory and motor structures in the brain display many features of optimally designed systems. Similarly, the brain mechanisms responsible for action selection might adopt optimal computational strategies that can be dynamically adjusted based on expected rewards. Recently, signals related to some elements in reinforcement learning algorithms have been identified in various brain areas, such as brainstem dopamine neurons signaling reward prediction errors. The proposed research program will bring together formal models (game theory and reinforcement learning), electrophysiology in behaving primates, and biophysically-based computational modeling of large- scale cortical networks. The proposed studies will (1) further develop the primate paradigm of a decision- making task based on competitive games, (2) examine the activity of single neurons in several key areas in the frontal lobe to identify neural basis of computational steps in dynamic decision-making, (3) develop a biophysically-based cortical network model for dynamic decision-making by implementing reward-based synaptic learning rules, (4) examine possible mechanisms responsible for the randomness of choice behavior, such as irregular spike activity and stochasticity in the synaptic learning rules, and (5) investigate the possibility that a distinct neural network is endowed with the ability of cross-trial temporal integration to estimate expected reward through experience.

DESCRIPTION (provided by applicant): Traditionally, individual modules of visual systems, such as perception of form, motion, and color have been studied separately. Increasingly, however, it is being appreciated that such visual functions are closely tied to other modules in the brain underlying learning, memory, and decision-making. For example, representation of the visual system is plastic and shaped by experience. In addition, although early visual pathway is characterized by its massive parallelism, the capacity of its working memory representation is severely limited, as dramatically demonstrated by the phenomenon of change blindness. Therefore, our understanding of visual functions must be complemented by studies of their relationship with working memory. Finally, the process of decision-making is a key to understanding how the sensory inputs are converted to behavioral responses. We propose to hold a Symposium at the Center for Visual Science at the University of Rochester in June, 2004, on the topic of Adaptive Representation and Control in Vision. The goal of the symposium is to facilitate the interactions among vision scientists and researchers of cognitive processes closely related to vision, such as perceptual learning, working memory, decision-making, and cortico-cortical interactions. In the tradition of past CVS Symposia, we will bring recent developments in these fundamentally important topics to a broader audience than that captured by more specialized meetings. We also wish to bring together speakers from a wide variety of areas in order to promote interactions between groups of investigators from diverse areas. This symposium will provide an exciting vehicle for students and investigators in vision and cognitive sciences to discuss and debate the values of various new paradigms in these rapidly expanding areas of research.

Project 3. Interactions in the Corticostriatal Network: Disruption of basal ganglia function leads to severe motor and cognitive deficits in various clinical conditions. Although the exact nature of basal ganglia functions remains elusive, recent progress in the study of basal ganglia anatomy and physiology have generated specific computational theories that can be tested in behaving animals. Because the main inputs and outputs of the basal ganglia arise from multiple cortical areas, the basal ganglia are likely to play a major role in the control of cortical information processing. This project will focus on the interactions between the prefrontal cortex and the striatum by simultaneously recording single-unit and field-potential activity from multiple sites in these two structures. The proposed experiments will use monkeys trained to perform reward-based learning and decision-making tasks. The first experiment will test the hypothesis that the convergence of cortical inputs and reward signals conveyed by the dopamine neurons provides a substrate in the striatum to generate signals necessary to bias the process of response selection. The second experiment will determine whether the role of the basal ganglia is limited to a situation in which a stable pattern of response bias can be established, or whether the neurons in the striatum are also involved in a dynamic decision-making task where the animal is required to avoid a habitual pattern of responses. Finally, the neural recordings obtained in these two experiments will be analyzed extensively using a wide range of analytical techniques to determine whether the temporal structure in the activity of striatal neurons is systematically influenced by synchronized oscillations in the cortical inputs. Overall, the results from these experiments will advance our understanding of the nature of communications between the cortex and the basal ganglia. Project 3 will complement other projects examining the role of the basal ganglia, motor cortex, supplementary motor area (Project 4) and ventral premotor cortex (Project 5) in selecting responses from among discrete alternatives, contributing to our shared goal of understanding how interactions among motor structures contribute to the selection and execution of coordinated eye, head and hand movements.

This project explores the evolutionary basis of human irrational decision-making. In the past few decades, behavioral economics has observed that people demonstrate a number of systematic decision-making biases in both the laboratory and the field. To date, however, very little research has explored the origins of such biases. Do economic biases reflect learned behaviors that result from particular market experiences, or are these biases the result of more ingrained tendencies, ones that result from evolved and possibly innate mechanisms? This proejct investigates the evolutionary origins of human economic behaviors, including persistent biases such as gain-loss asymmetry and reference dependence. To do so, this project conducts a series of economic studies with a novel set of subjects, non-human primates. Although non-human primates lack market conditioning, they too face a variety of distinctly economic problems, such as coping with the scarcity of goods, assigning the limited resources necessary to acquire them, and managing risks inherent in their native ecology.

It is possible, then, that certain aspects of economic decision-making will follow from evolutionarily-specialized mechanisms designed to deal with primitive resource conditions that foreshadowed modern economies. This project introduces a fiat currency and trade to a captive group of primates and then verifies the results of these market experiments. The project then uses this method in a series of studies to explore primate preferences over a range of economic problems typically used to extract data describing human decision-making. These studies will be the first of their kind to determine whether human-like standards of rationality are present in our close primate relatives and, if so, whether homologous psychological architectures can explain rational choice in both species. Demonstrating that primates either do or do not share economic biases with humans will have implications for which policy interventions are likely to overcome such biases in human markets and can be used to develop new behavioral measures for future neuroscientific and clinical investigations of human economic biases.

Stress can be defined as any real or perceived threat to the psychological and physical integrity of an individual, and despite its short-term adaptive value, can cause behavioral, emotional, cognitive, and immune changes in affected individuals that are potentially harmful. Furthermore, stress can indirectly lead to other negative health consequences by altering the behavioral strategies of the individuals. For example, it has been demonstrated that stress can lead to poor diet, obesity, and substance abuse. It has been hypothesized that such maladaptive behaviors might arise as a result of stress-induced weakening of the prefrontal cortical functions. Previous studies of stress-induced changes in prefrontal functions showed that excessive dopamine and norepinephrine release in prefrontal cortex contributed to working memory impairment. However, the effects of stress on the decision-making functions of the prefrontal cortex have not been studied. In particular, impulsive choice is a strong predictor for many addictive behaviors, such as smoking and substance abuse, that are exacerbated by stress. Therefore, the effect of stress on impulsivity and prefrontal functions will be the focus of this project. In the proposed study, impulsivity will be quantified by the animal's tendency to choose a small but immediate reward rather than a large but delayed reward in an inter-temporal choice task. First, it will be tested whether impulsivity is increased by a pharmacological stressor, FG7142. Second, the role of dopaminergic and/or noradrenergic pathways in the regulation of impulsive choice behavior will be tested using receptor specific agonists and antagonists. Third, we will examine how these pharmacological manipulations affect the neural process of discounting the value of delayed reward by recording the single-neuron activity of neuronal ensembles in the prefrontal cortex with and without the same pharmacological manipulations. Fourth, using the combination of iontophoresis and single-neuron recording, it will be tested whether prefrontal functions related to impulsive choice behaviors are altered by dopamine and norepinephrine through their direct actions on the prefrontal micro-circuitry. Overall, the results from these experiments will provide a critical piece of information regarding the cellular basis of stress-mediated impairments in adaptive behavioral strategies and normal prefrontal functions, and thereby contribute to the development of new preventive and therapeutic measures against stress-mediated disease processes.

Drug addiction can be characterized by the impaired abilities to choose the course of action that is likely to lead to the best long-term consequences for the affected individual and society. In particular, two different forms of impulsivity might both contribute to the initiation of addictive behaviors and become exacerbated by them. First, compared to healthy controls, drug addicts tend to show stronger preference for an immediate reward over a more delayed but larger reward. This is referred to as choice impulsivity. Second, drug abusers tend to initiate actions prematurely before the sufficient evidence for such actions is accumulated. This is referred to as response impulsivity. A main goal of Project 3 is to invesfigate how these two types of impulsivity might be mediated by the striatum in primates, and to test whether cocaine-induced changes in striatal functions might account for changes in the impulsive choices and responses. To investigate choice impulsivity, animals will be trained to perform an inter-temporal choice in which they will choose between two different rewards that differ in their magnitude and immediacy. Their choice impulsivity will be estimated from the steepness of a discount function that describes how the subjective value of reward decreases with the reward delay. Response impulsivity will be measured with a stop-signal task in which the animal receives an instrucfion to produce an eye movement towards a peripheral target in every trial but is also required to cancel such movement when a second "stop" signal is presented. The time necessary to cancel the unwanted movement will be estimated to quantify response impulsivity. We will then characterize and compare the activity of individual neurons in the caudate nucleus and ventral striatum while the animals perform each of these two tasks before and after cocaine exposure. Specifically, we will test whether the striatal acfivity related to the reward magnitude and delay will be differenfially affected by cocaine exposure. We will also test whether the striatal activity related to the preparation and cancellation of visually guided eye movements are modified by cocaine. The results from these experiments will elucidate the role of striatum in drug addiction.

DESCRIPTION (provided by applicant): Patients with lesions in the orbitofrontal cortex display characteristic behavioral deficits in various decision- making tasks. They are impaired in revising their behavioral strategies when the previously advantageous actions no longer produce desirable outcomes and integrating multiple dimensions of the decision outcomes to make optimal choices. These behavioral changes resemble the symptoms of drug addicts who are unable to discontinue the use of abused substance that are no longer pleasurable and lead to negative consequences. Dysfunctions of the orbitofrontal cortex are also implicated in many mood and anxiety disorders. In addition, patients with orbitofrontal lesions are unable to take into consideration hypothetical outcomes from unchosen actions to improve their decision-making strategies. Despite this wide clinical implication of the orbitofrontal dysfunctions, the nature of specific computational steps embodied in the orbitofrontal cortex and disrupted in many mental disorders is unknown. Studies proposed in this application will investigate how multiple reward- related parameters and various decision variables are encoded and integrated by individual neurons of the primate orbitofrontal cortex. First, we will test whether the neurons in the orbitofrontal cortex will integrate multiple reward parameters during an inter-temporal gambling task in which the magnitude, delay, and uncertainty of reward available from each option is systematically manipulated. Second, we will investigate whether neurons in different subdivisions of the orbitofrontal cortex tend to specialize in propagating the signals related to positive and negative outcomes. This will be tested in a token-based decision-making task in which the animal's choice behaviors are reinforced and punished by the delivery and removal of conditioned reinforcers, respectively. Finally, we will also test whether the information about the hypothetical outcomes from unchosen actions are reflected and integrated with the signals related to the animal's choices in the activity of orbitofrontal neurons. The results from these experiments will elucidate how the signals originating from different sources are integrated and transformed in the orbitofrontal cortex so that they can be used directly to guide the animal's choices. Accordingly, the proposed studies will contribute to prevention and more efficient treatment strategies for substance abuse and other mental disorders that involve the orbitofrontal cortex.

DESCRIPTION (provided by applicant): Dysfunctions of the basal ganglia (BG), such as Parkinson's disease, obsessive-compulsive disorder, and substance abuse, produce a number of motor and cognitive deficits. Nevertheless, current therapeutic interventions are mostly empirical, and their underlying mechanisms remain poorly understood. This reflects our poor knowledge about how different subdivisions of the basal ganglia interact so that their outputs can adaptively modulate the activity of neurons in their downstream structures. So far, almost all influential models of the BG have proposed that they are involved in choosing appropriate actions (action selection) and altering the tendencies to choose different actions according to their previous outcomes (reinforcement learning). Nevertheless, how these functions are implemented across parallel anatomical pathways through the BG remains crudely understood, because only a small number of physiological studies have systematically compared the activity of neurons across different subdivisions of the BG during behavioral tasks designed to probe specific cognitive processes. Studies proposed in this application will investigate how signals distributed in three major components of the BG contribute to action selection and reinforcement learning. Specifically, we will focus on the output nuclei of the basal ganglia, namely, the internl segment of the globus pallidus (GPi) and the substantia nigra pars reticulata (SNr), in addition to the external segment of the globus pallidus (GPe) that is believed to exert substantial influences on all other components of the BG. The comparison of neural signals across different compartments of the BG is essential for understanding the nature of intra- and trans- basal-ganglia signal processing, including the role of the direct and indirect pathways. In our experiments, we will first test whether estimates for the outcomes expected from alternative actions are dynamically and continually updated in the GPe and GPi/SNr during a decision-making task. In particular, whether they show enhanced activity related to unexpected changes in the expected outcomes will be tested. Second, we will also test whether the signals related to rewards and penalties are encoded differentially across different subdivisions of the BG. The results from these studies will lay important foundations for developing more efficient treatments for a number of mental disorders resulting from BG dysfunctions.

? DESCRIPTION (provided by applicant): All cognitive operations including perceptual and motor processes unfold in time, and actions lead to successful outcomes only when they are executed at appropriate times. However, controlling multiple actions and monitoring their outcomes over a relatively long temporal interval poses challenging computational problems for the brain. Moreover, the ability to utilize temporal information is impaired in various psychiatric and neurological disorders, including attention-deficit/hyperactivity disorder (ADHD), Parkinson's disease, and substance abuse. Despite such broad theoretical and clinical significance, previous neurobiological studies on timing behavior and temporal decision-making have focused narrowly on the brain functions related to a single temporal interval or a unitary reward. The main goal of this proposal is to gain novel insights into the neural mechanisms capable of concurrently monitoring multiple temporal intervals and evaluating the value of multiple outcomes in a sequence. First, we will investigate the anatomical specificity of timing signals in the medial and lateral areas of the primate prefrontal cortex using a task that requires the animal to plan different actions according to two concurrent temporal intervals. We will test the hypothesis that the medial prefrontal cortex plays a special role in creating the internal timing signals from transient sensory events and transforming them into motor responses. Second, we will study the mechanism in the fronto-parietal network for integrating the values of multiple rewards in a sequence while the animal chooses between two separate temporally extended reward sequences. Specifically, we hypothesize that the flexible transformation of signals related to reward magnitude between the posterior parietal cortex and prefrontal cortex underlies the process of integrating the values of individual rewards into a single decision variable. The results from these two experiments will advance our knowledge about how the brain handles the timing information about multiple events efficiently, and lay the foundation for understanding the nature of clinical conditions with impaired abilities to process temporal information.

? DESCRIPTION (provided by applicant): The NMDA receptor (NMDAR) antagonist, ketamine, is currently in experimental use for the treatment of severe, intractable depression. Emerging clinical data indicate that intranasal ketamine administration can produce relief within minutes (5~40min), with fewer cognitive side effects than systemic ketamine injections. The proposed research will use a non-human primate paradigm to study the cellular and circuit mechanisms underlying the effects of intranasal vs. injected ketamine administration in prefrontal cortical (PFC) circuits immediately relevant to depression and cognition. We have observed persistent neuronal activity in medial PFC related to aversive events in a decision-making task, which may contribute to negative affective states and be abnormally heightened in depression. We will test the hypothesis that ketamine rapidly erodes the persistent representations of loss and punishment generated by medial prefrontal neurons, similar to ketamine's ability to erode persistent representations of visual space by dorsolateral prefrontal cortical (dlPFC) neurons. We hypothesize that the disruption of neural circuits representing loss may underlie the ultra-rapid effects of intranasal ketamine in patients (within minutes), while spinogenesis in higher order PFC regions may contribute to the more sustained anti-depressant actions (hours to days). As intranasal inhalation delivers drug directly to the brain through the holes in the cribiform plate, we further hypothesize that the medial PFC regions in the direct trajectory of intranasal ketamine may be more affected than the dlPFC neurons that are more distant. Such data would help to explain why intranasal administration has a more rapid onset with fewer cognitive side effects than ketamine injections. Aim 1 will compare the effects of intranasal vs. intramuscular administration of sub-anesthetic doses of ketamine on performance of the decision-making vs. spatial working memory tasks. We predict that ketamine will attenuate the effects of previous punishment on decision-making, allowing a more resilient behavioral response, and that intranasal administration will preferentially influence this behavior, while IM injections will have more global effects on performance in both tasks. Aim 2 will compare the effects of intranasal vs. intramuscular administration of ketamine on persistent neuronal firing in medial PFC regions relevant to depression (BA24, BA25 and dmPFC) compared to the dlPFC. We predict that ketamine will rapidly erode persistent representations of loss in medial PFC areas. We further hypothesize that intranasal administration will have more targeted effect on medial PFC neurons, while IM ketamine will alter neuronal firing in both medial and lateral PFC areas. Finally, Aim 3 will test whether the effects of ketamine are mediated by local actions in the medial PFC using iontophoretic application of ketamine and other more selective NMDAR NR2A vs. NR2B antagonists directly onto medial PFC neurons, similar to previous experiments in the dlPFC. The results from these experiments will provide important insights into the cellular and circuit mechanisms underlying the rapid anti-depressant effects of ketamine.

Humans and other animals can choose their actions using multiple learning algorithms and decision­ making strategies. For example, habitual behaviors adapted to a stable environment can be selected using so-called model-free reinforcement learning algorithms, in which the value of each action is incrementally updated according to the amount of unexpected reward. The underlying neural mechanisms for this type of reinforcement learning have been intensively studied. By contrast, how the brain utilizes the animal's knowledge of its environment to plan sequential actions using a model-based reinforcement learning algorithm remains unexplored. In this application, PIs with complementary expertise will investigate how different subdivisions of the primate prefrontal cortex contribute to the evaluation and arbitration of different learning algorithms during strategic planning in primates, using a sequential game referred to as 4-in-a­ row. Previous studies have revealed that with training, humans improve their competence in this game by gradually switching away from a model-free reinforcement learning towards a model-based reinforcement learning in the form of a tree search. In the first set of experiments, we will train non-human primates to play the 4-in-a-row game against a computer opponent. We predict that the complexity of the strategic planning and the opponent's move violating the animal's expectation will be reflected in the speed of animal's action and pupil diameters. Next, we will test how the medial and lateral aspects of prefrontal cortex contribute to the evaluation and selection of different learning algorithms during strategic interaction between the animal and computer opponent. We hypothesize that the lateral prefrontal cortex is involved in computing the integrated values of alternative actions originating from multiple sources and guiding the animal's choice, whereas the medial prefrontal cortex might be more involved in monitoring and resolving the discrepancies of actions favored by different learning algorithms. The results from these experiments will expand our knowledge of the neural mechanisms for complex strategic planning and unify various approaches to study naturalistic behaviors. By taking advantage of recent advances in machine learning and decision neuroscience, proposed studies will elucidate how multiple learning algorithms are simultaneously implemented and coordinated via specific patterns of activity in the prefrontal cortex. The results from these studies will transform the behavioral and analytical paradigms used to study high-order planning and their neural underpinnings in humans and animals.

Humans and other animals can choose their actions using multiple learning algorithms and decision­ making strategies. For example, habitual behaviors adapted to a stable environment can be selected using so-called model-free reinforcement learning algorithms, in which the value of each action is incrementally updated according to the amount of unexpected reward. The underlying neural mechanisms for this type of reinforcement learning have been intensively studied. By contrast, how the brain utilizes the animal's knowledge of its environment to plan sequential actions using a model-based reinforcement learning algorithm remains unexplored. In this application, PIs with complementary expertise will investigate how different subdivisions of the primate prefrontal cortex contribute to the evaluation and arbitration of different learning algorithms during strategic planning in primates, using a sequential game referred to as 4-in-a­ row. Previous studies have revealed that with training, humans improve their competence in this game by gradually switching away from a model-free reinforcement learning towards a model-based reinforcement learning in the form of a tree search. In the first set of experiments, we will train non-human primates to play the 4-in-a-row game against a computer opponent. We predict that the complexity of the strategic planning and the opponent's move violating the animal's expectation will be reflected in the speed of animal's action and pupil diameters. Next, we will test how the medial and lateral aspects of prefrontal cortex contribute to the evaluation and selection of different learning algorithms during strategic interaction between the animal and computer opponent. We hypothesize that the lateral prefrontal cortex is involved in computing the integrated values of alternative actions originating from multiple sources and guiding the animal's choice, whereas the medial prefrontal cortex might be more involved in monitoring and resolving the discrepancies of actions favored by different learning algorithms. The results from these experiments will expand our knowledge of the neural mechanisms for complex strategic planning and unify various approaches to study naturalistic behaviors. By taking advantage of recent advances in machine learning and decision neuroscience, proposed studies will elucidate how multiple learning algorithms are simultaneously implemented and coordinated via specific patterns of activity in the prefrontal cortex. The results from these studies will transform the behavioral and analytical paradigms used to study high-order planning and their neural underpinnings in humans and animals.