Nathaniel D. Daw - US grants

DESCRIPTION (provided by applicant): A striking range of mental disorders, from OCD to schizophrenia, is accompanied by aberrant decision-making and also by dysfunction in the dopamine system and its targets in the forebrain. Although celebrated computational work posits roles for this system together with the posterior parietal cortex in learning and decision-making for simple choice problems, it requires a tremendous leap of faith to imagine how these simple computational mechanisms can be "scaled up" from the laboratory to address real-world human behavior of the sort that is clinically problematic for patients with these disorders. One understudied aspect of this problem is the high dimensionality of the space of candidate actions, notably the involvement of multiple effectors such as hands and eyes. This project proposes a theoretical framework for more realistic learning and decision problems involving multiple effectors, and leverages it in experiments probing how the brain copes with learning and decision-making in these cases. The core idea is that the brain should divide-and-conquer: treating, e.g., hand and eye movements independently to simplify learning when their consequences are independent, but that it must evaluate actions jointly across effectors when this is not the case. Learning tasks manipulating this independence are used to: (1) test whether humans and animals learn to solve decision problems by separating or coordinating effector choices to efficiently harvest rewards;these tasks are combined with electrophysiological recordings and fMRI to (2) test whether separate or conjoint neural value maps are maintained for action values across effectors, as appropriate to the problem;and multiarea recordings are used to (3) test whether coordinated choices increase neural interactions between effector-specific motor maps. The work makes innovative use of computational theory for experimental design and analysis, in order to connect experimental observations across species, measurement types (spiking, local field potentials, fMRI), and scales (neuronal, systems). It also introduces a new laboratory microcosm for the computations needed to scale up existing decision theories toward clinically relevant real-world behaviors. In principle, quantitative theories of the brain's decision and learning systems hold important promise for the numerous serious mental illnesses that center around these systems, such as improved procedures for diagnosis or screening candidate treatments. This project aims to "scale up" such theories -- which are, in practice, too simple to deliver on this promise -- toward explaining the interacting neural circuits that control realistic behaviors more like those that are problematic for patients with mental illnesses.

DESCRIPTION (provided by applicant): What aspects of previous experiences guide decisions? Much research concerns how the brain computes the average, over many experiences, of rewards received for an option. But such a summary - produced by prominent models of dopaminergic incremental learning- is chiefly useful for repetitive tasks. Much less is understood about how the brain can flexibly evaluate new or changing options in more realistic tasks, which must rely on less aggregated information. This application argues that this is fundamentally a function of memory, so this project looks to the brain's memories for the most individuated experiences - episodes - to seek new computational, cognitive and neural mechanisms that could support more flexible decisions. The overarching hypothesis is that episodic memory, supported by the hippocampus, plays a central role in guiding flexible decision making and complements the wellknown role of dopaminergic and striatal systems in incremental learning of value. What is the intellectual merit of the proposed activity? By connecting the computational neuroscience of decision making with the cognitive neuroscience of memory, and bringing together collaborators from each area, this project promises to shed light on both areas. This is because the neural mechanisms supporting episodic memory are well studied, but less so their contribution to adaptive behavior. Computationally, episodic memories can support a family of learning algorithms that draw on sparse, individual experiences, such as Monte Carlo and kernel methods. These suggest novel, plausible hypotheses for how the brain solves more realistic decision problems, and in particular how it implements goal-directed or model-based choices. The proposed studies aim to differentiate the contributions of incremental and episodic learning to value-based decisions, and test to what extent episodic memories contribute to decisions previously identified as model-based. Our hypotheses are tested fitting computational models to neural activity from functional MRI experiments in humans, and also to choice behavior in healthy individuals compared to patients with isolated damage to specific neural systems. This combination of computational, neuroimaging and neuropsychological approaches permits finely tracing the trial-by-trial dynamics of learning as reflected both in brain activity nd behavior, and also testing the causal role of particular brain regions in these same processes. What are the broader impacts of the proposed activity? A striking range of psychiatric and neurological disorders, including Parkinson's disease, schizophrenia and eating disorders, are accompanied by aberrant decision-making and by dysfunction in circuitry central to this proposal, such as striatal and fronto-temporal mechanisms. But understanding such dysfunction requires a better understanding of how each of these circuits separately influences decisions. A focus on untangling multiple decision systems is particularly pertinent to disorders such as drug abuse, which is hypothesized to center on the compromise of incremental reinforcement mechanisms that may support more habitual actions and underlie the compulsive nature of such diseases. At the same time, drugs may also weaken or compromise more deliberative or goal-directed choice systems that might otherwise be able to support more advantageous decisions. Formally understanding the roles played by both of these influences, and how they interact, promises to improve the conceptualization, diagnosis, and treatment of these and other disorders. The proposed program also provides unique opportunities for training and education. By integrating multiple core tools of systems and cognitive neuroscience (computational modeling, functional imaging, patient studies, behavioral analyses), students in the labs of both PIs are trained in different approaches to a unified research question, preparing them to be effective scientists in a more interdisciplinary future. Components of this training will also be extended to undergraduate and high school student populations through existing programs at both NYU and at Columbia and through outreach to New York area schools. This project will also help promote broader representation of minorities in science, including women. As a female neuroscientist with many women trainees in her laboratory, PI Shohamy serves as a role model and the collaborative project facilitates training for women in computational neuroscience, an area in which women are particularly underrepresented.

? DESCRIPTION (provided by applicant): Overview: Among the most celebrated success stories in computational neuroscience is the discovery that many aspects of decision-making can be understood in terms of the formal framework of reinforcement learning (RL). Ideas drawn from RL have shed light on many behavioral phenomena in learning and action selection, on the functional anatomy and neural processes underlying reward-driven behavior, and on fundamental aspects of neuromodulatory function. However, for all these successes, RL-based work is haunted by an inconvenient truth: Standard RL algorithms scale poorly to large, complex problems. If human learning and decision-making are driven by RL-like mechanisms, how is it that we cope with the kinds of rich, large-scale tasks that are typical of everyday life? Existing research in both psychology and neuroscience hints at one answer to this question: Complex problems can be conquered if the decision-maker is equipped with compact, intelligently formatted representations of the task. This principle is seen in studies of expert play in chess, which show that chess masters leverage highly integrative internal representations of board configurations; in studies of frontal and parietal lobe function, which have revealed receptive fields strongly shaped by task contingencies; and studies on the hippocampus, which point to the role of this structure in supporting a hierarchically organized 'cognitive map,' of task space. Not coincidentally, the critical role of representation has come increasingly to the fore in RL-based research in machine learning and robotics, with growing interest in techniques for dimensionality reduction, hierarchy and deep learning. The present project aims toward a systematic, empirically validated account of the role of representation in supporting RL and goal-directed behavior at large. The project brings together three investigators with complementary expertise in cognitive and computational neuroscience (Botvinick, Gershman) and machine learning and robotics (Konidaris). Together, we propose an integrative, interdisciplinary program of research, applying behavioral and neuroimaging work with human subjects, computational modeling of neurophysiological and behavioral data, formal mathematical work and simulations with artificial agents. The proposed studies are diverse in theme and method, but work together toward a theory that is both formally grounded and empirically constrained. At a more concrete level, our research focuses on four specific classes of representation, considering the computational impact of each for RL, as well as the relevance of each to neuroscience and human behavior. As detailed in our Project Description, these include (1) metric embedding, (2) spectral decomposition, (3) hierarchical representation and (4) symbolic representation. In addition to investigating the implications of each of these four forms of representation individually, we hypothesize that they fit together into a tiered system, which works as a whole to support the sometimes competing demands of learning and action control. Intellectual Merit (provided by applicant): Understanding how representational structure impacts learning and decision making is a core challenge in cognitive science, behavioral neuroscience and, artificial intelligence. Success in establishing a computationally explicit, empirically validated theory in this area, with a specific focus on the role of representation in R, would represent an important achievement with wide repercussions. The strategy of leveraging conceptual tools from machine learning to investigate human behavior and brain function can offer considerable scientific leverage, as our own previous research illustrates. The proposed work is motivated by and builds upon established lines of research, bringing these together in order to capitalize on opportunities for synergy. In addition to answering specific empirical and computational questions, the proposed work aims to open up new avenues for future research in an important area of inquiry. Broader Impact (provided by applicant): The proposed work lies at the crossroads of neuroscience, psychology, artificial intelligence and machine learning, and promises to advance the growing exchange among these fields. The project brings together investigators with contrasting disciplinary affiliations, with the explicit goal of bridging between intellectual cultres. The proposed work is likely to find a wide scientific audience, given its relevance to cognitive and developmental psychology, behavioral, cognitive and systems neuroscience, and behavioral economics. However, the work is likely to be of equal interest within artificial intelligence, machine learning, and robotics, where a current challenge is precisely to understand how representation learning can allow RL to scale up to large problems. Representational approaches to RL are already of intense interest within industry, where the present investigators have a record of active engagement. The topic of the proposed work has applicability in other areas as well, including education and training, and military and medical decision support. The plan for the project has a robust training component at both graduate and postdoctoral levels, with a commitment to fostering involvement of underrepresented minorities, as well as international engagement.

The decisions we make are shaped by memories of our previous experiences. Indeed, what decision you make may ultimately depend on which memories your brain accesses, and which ones you neglect, in contemplating a potential action's outcome. This project aims to measure memory access in support of choices, using functional neuroimaging, so as to study which memories are accessed when, and how retrieving these memories affects the choices people make, either immediately or later on. Understanding these processes will lay the foundation for a better, more unified understanding of many diverse phenomena affecting choices - planning, when habits arise, the role of dreams, and the impacts of advertising. This could also improve our understanding of maladaptive choice in various disorders, such as rumination, compulsion, and craving. The experiments also aim to examine how manipulating the structure of previous experience affects these memory-access patterns, and ultimately choices. In addition to its scientific aims, the project aims to train young scientists in an interdisciplinary range of techniques, combining computational and cognitive neuroscience, and to serve diversity especially by facilitating training of women in these areas.

Actions can be separated from their consequences by many steps in space and time. Anticipating these consequences so as to choose the best actions requires integrating memories of multiple intermediate events, which often were not originally experienced together. But so far there has been a lack of a principled and unified account of which memories are accessed, when, and which are neglected, to support value-based decisions. This project aims to test a recent computational theory that formalizes the ways in which particular memories are accessed and integrated to evaluate options, and the consequences for choice. The overarching hypothesis is that the brain sequentially integrates multiple memories for separate experiences either retroactively or prospectively, prioritizing the most valuable ones depending on the statistics of previous experience. The project will test this hypothesis using functional Magnetic Resonance Imaging in humans that will be engaged in solving several reinforcement-learning tasks. The project will take advantage of category-specific visual activity to measure memory access at different points during acquisition and deliberation, compare these patterns to subsequent choices, and test whether manipulations of the statistical structure of experience affects both memory access and the resulting choices.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Project Summary The differentiation between positive and negative valence is central to psychiatry. A seemingly categorical distinction between the drive toward rewards vs. the effort to avoid punishment appears central to many symptoms of psychiatric dysfunction and is evident in both how diagnostic categories are delineated and in the definition of cross-diagnostic constructs in RDoC. However, while there has been major progress in understanding how reward drives learning and actions and the underlying neural mechanisms, there has been much less progress in understanding the mechanisms by which loss and punishment affect behavior. Indeed, there has been continued controversy about whether the neural mechanisms of reward and loss are dissociable at all. Studies of the neural bases of reward seeking vs. loss avoidance have yielded mixed results, manifested both in inconsistent findings about shared vs. separate neural circuitry, and in surprising results in psychiatric populations, for instance showing reward processing abnormalities in psychiatric conditions that appear at face value to be driven by avoidance (e.g. OCD and anxiety). This has made it virtually impossible to address the critical question of defining valid measurements for reward seeking vs. loss avoidance separately, let alone for understanding the balance between them and their relation to other dimensional constructs and psychopathology. Here we address this challenge. We build on a computational framework that resolves the inconsistency in existing results by formalizing how avoiding a loss can ? in certain circumstances and in some people ? be reframed as a reward. Here we advance the hypothesis that using computational methods for quantifying and isolating this subjective reframing will allow us better to disentangle the relative, covert contributions of reward seeking vs. loss avoidance, and clarify their neural underpinnings. We propose to test this hypothesis by rigorously assessing the validity of the resulting measures (compared to simpler measures of overt reward and loss behavior) across tasks, measures, and test-retest replications. In particular, we address two specific aims. First, we seek to compare neural and behavioral measures of reward seeking and loss avoidance across tasks and participants using computational models and functional MRI in a large sample of participants. Second, we seek to examine individual differences in reward seeking and loss avoidance learning and their relationship to dimensions of psychiatric symptomatology using a large online sample. Both aims make use of two parallel and complementary experimental tasks which each test reward seeking, loss avoidance, and the extent to which the balance between the two is affected by differences in baseline expectations of reward or loss. Together, these studies offer an integrative computational framework to test the construct validity of measures of reward seeking and loss avoidance, the relationship between them, the new construct of their relative reframing, and how individual differences in these constructs are manifest across the population in brain and behavior.