John P. O'Doherty - US grants

In the course of everyday life, people are frequently faced with decisions between different goals. Often these involve choices between different types of rewards. For example, should I spend extra hours at work to get that salary bonus, or should I spend that time instead to be with my family? In order to develop an understanding of how the brain computes decisions between different types of reward it is necessary to first determine how each of these different types of reward are represented in the brain. The goals of this project are to determine whether different rewards are represented in different brain areas, and whether there also exists a system that responds similarly to different types of reward. With support from the National Science Foundation, Dr. John O'Doherty and colleagues at the California Institute of Technology will address these questions by using brain imaging techniques (functional magnetic resonance imaging) to probe neural responses to rewards and punishments in a number of key brain regions known to be involved in processing of emotional responses and rewards, including the amygdala, orbitofrontal cortex, ventral striatum and anterior insula. A series of experiments will be conducted during which human volunteers will be presented with two different types of reward and punishment: juice (pleasant and aversive) and money (wins and losses), interleaved within the same experiments. These responses will then be compared directly to test for regions responding to both types of reward or punishment, as well as to determine regions that respond exclusively to one or other type. Brain responses will be measured not only to the receipt of reward, but also to their expectation.

This work will result in a more comprehensive picture of reward and punishment representations in the human brain. The research could also provide insights into the mechanisms underlying complex decision making behaviors that depend on integration of different types of reward information. The findings generated by this research could help elucidate the fundamental learning mechanisms that underlie all motivated behaviors. Such findings could have a significant impact on fields outside cognitive neuroscience such as economics and decision theory. This research could also lead to the development of novel techniques to help people make better decisions, or improve learning and skill acquisition through the use of reinforcement. The funding from this application will be used to support a new research group at Caltech which can provide research training opportunities for undergraduate, graduate and post-doctoral trainees in cognitive neuroscience and brain imaging.

[unreadable] DESCRIPTION (provided by applicant): The ability to orient toward specific goals in the environment and control actions flexibly in pursuit of those goals is a hallmark of adaptive behavior. Instrumental conditioning is the simplest form of such behavior in which an animal or human learns to perform an action or sequence of actions in order to obtain reward or avoid punishment. Instrumental conditioning is composed of two distinct components: a goal-directed and a habit-learning component. In goal-directed learning, associations are formed between a given action and the goal-state (future reward or punishment). Instrumental behavior under goal-directed control can be altered rapidly following a change in the action-reward contingencies, or a decrease in the reward value of the goal itself. In habit learning, associations are made between the context (configuration of cues in the environment) and the given action, without encoding the goal itself. In contrast to goal-directed learning, habit learning is inflexible and leads to compulsive behavior. Once an action has become a habit (which happens over the course of learning), it is liable to be performed irrespective of the current value of the goal state. The aim of this project is to determine the neural substrates of goal-directed learning and habit learning in the human brain, and characterize the process by which a goal-directed action becomes transferred to a habit. This will be accomplished by scanning human subjects using functional magnetic resonance imaging (fMRI) to measure neural responses during performance of simple instrumental conditioning tasks. We will use specific task manipulations, track changes in fMRI signals over the course of learning, and apply formal computational models to fMRI data in order to uncover these different components in instrumental conditioning. This R03 application is to support the initial development of a research program by the principal investigator who has just started in a faculty position as a prelude to a subsequent application under the R01 mechanism. Uncovering the neural mechanisms mediating goal-directed and habit learning could have important implications for understanding how some behaviors become "habitized" - compulsive and difficult to change using "willpower", as is the case in obsessive compulsive disorders, eating disorders, compulsive gambling and drug addiction. This research could ultimately be relevant in developing treatments for such conditions in which habitized or compulsive behaviors are returned to goal-directed control. [unreadable] [unreadable] [unreadable]

[unreadable] DESCRIPTION (provided by applicant): The ability to orient toward specific goals in the environment and control actions flexibly in pursuit of those goals is a hallmark of adaptive behavior. Instrumental conditioning is the simplest form of such behavior in which an animal or human learns to perform an action or sequence of actions in order to obtain reward or avoid punishment. Instrumental conditioning is composed of two distinct components: a goal-directed and a habit-learning component. In goal-directed learning, associations are formed between a given action and the goal-state (future reward or punishment). Instrumental behavior under goal-directed control can be altered rapidly following a change in the action-reward contingencies, or a decrease in the reward value of the goal itself. In habit learning, associations are made between the context (configuration of cues in the environment) and the given action, without encoding the goal itself. In contrast to goal-directed learning, habit learning is inflexible and leads to compulsive behavior. Once an action has become a habit (which happens over the course of learning), it is liable to be performed irrespective of the current value of the goal state. The aim of this project is to determine the neural substrates of goal-directed learning and habit learning in the human brain, and characterize the process by which a goal-directed action becomes transferred to a habit. This will be accomplished by scanning human subjects using functional magnetic resonance imaging (fMRI) to measure neural responses during performance of simple instrumental conditioning tasks. We will use specific task manipulations, track changes in fMRI signals over the course of learning, and apply formal computational models to fMRI data in order to uncover these different components in instrumental conditioning. This R03 application is to support the initial development of a research program by the principal investigator who has just started in a faculty position as a prelude to a subsequent application under the R01 mechanism. Uncovering the neural mechanisms mediating goal-directed and habit learning could have important implications for understanding how some behaviors become "habitized" - compulsive and difficult to change using "willpower", as is the case in obsessive compulsive disorders, eating disorders, compulsive gambling and drug addiction. This research could ultimately be relevant in developing treatments for such conditions in which habitized or compulsive behaviors are returned to goal-directed control. [unreadable] [unreadable] [unreadable]

A common assumption is that in order to get individuals to perform better on a particular task, they should be rewarded in proportion to how well they perform. This reasoning is behind performance-based pay in the workplace. However, psychologists and economists have long recognized that this type of relationship between rewards and performance only holds up to a point: When rewards get too large, often performance for a skilled task decreases rather than increases, compared to performance levels for a more moderate reward. With funding from the National Science Foundation, Dr. John O'Doherty and colleagues at the California Institute of Technology are investigating why when a reward becomes especially large, individuals become very focused on the possibility of losing or failing to attain that reward. The researchers are studying whether the possibility of failure causes interference in the parts of the brain involved in performing a skilled act. The investigators are using functional magnetic brain imaging (fMRI) to measure brain activity while volunteers perform a skilled task that involves a sequence of careful hand movements for which they receive differing amounts of potential monetary rewards, ranging from small to relatively large. The researchers are investigating whether performance levels for the larger rewards decreases relative to that for medium incentive levels, and whether decreasing brain activity in an area called the "ventral striatum" is related to individual performance decrements. In this project, the investigators are using multiple methods to manipulate how much individuals are focused on obtaining a negative and/or losing outcome while they perform the task. The researchers assert that a) the more individuals are focused on losing, the greater their performance declines; whereas, conversely, the less individuals are oriented to the possibility of losing, the greater their performance increase and that b) differences in focus are related to differing patterns of activity in the brain.

Understanding why it is that performance sometimes decreases when the rewards (or stakes) are large has important implications in a number of areas. The knowledge gained could be used to design better schemes for incentivizing people to perform well on a work task, by minimizing the potential for performance deteriorations. More generally, many human endeavors demand high levels of performance under conditions of high stakes (high potential gain and high potential losses). Competitive sports are an obvious example, but similar scenarios likely arise in other high-pressure contexts, such as during surgery or the piloting of aircraft. A better understanding of how and why performance decrements occur could aid the development of new strategies to minimize their impact. Dr. O'Doherty participates in the public dissemination of research findings through a variety of media, including news papers and magazines, science blogs, and radio. The present project also contributes to the scientific training of undergraduate and graduate students, and the results of the research will be incorporated into courses taught at the undergraduate and graduate level.

DESCRIPTION (provided by applicant): "Characterizing habitual and goal-directed behavioral control systems in the human brain using computational and multivariate fMRI". PI: Dr. John P. O'Doherty Institution: California Institute of Technology PROJECT SUMMARY While much is now known about the behavioral and neural mechanisms underlying goal-directed and habitual learning in rodents, much less is known about the brain structures involved in encoding the associations that support these two types of learning humans, and even less is known about the neural computations underlying their implementation. Even more critically, almost nothing is known about the mechanisms governing the transition in behavioral control between these two systems in humans. This project seeks to address these critical gaps in knowledge. To achieve this we will combine sophisticated behavioral protocols, inspired by animal studies of instrumental conditioning, with state-of-the-art fMRI data analysis. We first deploy multivariate pattern analysis techniques in order to establish the nature of associative encoding in candidate brain structures for goal-directed and habit learning such as the vmPFC, anterior and posterior striatum and supplementary motor cortex. Next, we apply sophisticated computational models to our behavioral and fMRI data in order to establish the nature of the computations underlying the implementation of these forms of learning in these brain areas. Once a clearer understanding of the neural implementation of goal-directed and habitual learning has been achieved, we turn our attention to the factors governing habitization, and to the neural systems involved in mediating the control of the habitual and goal-directed systems over behavior. For this we will apply a novel experimental paradigm developed in our laboratory that can induce behavioral habitization rapidly in human volunteers without the need for cumbersome over-training or other impractical procedures hitherto used to induce habits in humans. By combining this procedure with fMRI we will be able to directly identify brain structures engaged when behavior is under habitual control. This project will provide new insights into how habitual and goal-directed learning is implemented in the brain, and shed light on the mechanisms underlying the control of these systems over behavior. Ultimately this research has the potential to lead to the development of new mechanisms for inducing habitual control in order to achieve the maintenance of adaptive and healthful behaviors. PUBLIC HEALTH RELEVANCE: A fundamental issue in the search for new behavioral treatments for disorders such as obesity, addiction and other psychiatric diseases is how to habitize behaviors that lead to healthful consequences, and thus ensure their maintenance without dependence on the effortful process of rational decision- making. In spite of substantial evidence from animal conditioning experiments, very little is known about the psychological and neural processes that underlie habitual performance in humans and that determine its dominance over goal- directed action selection. In this proposal we aim to shed light on how habits are processed in the brain, and to investigate the neural mechanism by which habits can come to control human behavior.

This Project investigates one of the most important ways in which humans learn to make sound decisions: through observing other people. Whereas Project 1 investigated how we value social and nonsocial stimuli (social reward processing), this Project 2 investigates how we learn from other people about rewards (observational learning). Do the same computations of decision values and experienced values that were investigated under Project 1 also come into play when we watch others? If so, do they recruit the same brain structures? Four Specific Aims systematically investigate how the mechanisms behind observational learning might contrast with the much better studied mechanisms behind learning from direct experience. The first Aim focuses on a structure thought to encode reward prediction errors in dopaminergic neurons that drive learning, the ventral striatum, and recordings in monkeys. The second Aim extends these studies to humans, using single-unit and LFP data from intracranial recordings, together with fMRI and focuses on the target structures of dopamine projections: amygdala and vmPFC. The third Aim asks what difference it might make exactly who we are observing: whether the person is competent or not, whether they are a friend or a stranger; this Aim looks at such modulation in the ventral striatum, the amygdala, and the ventromedial prefrontal cortex. The fourth Aim explores possible individual differences between the fMRI data obtained under Aims 1 and 2, and the psychological variables assessed under Core 3.

US-German Collaboration: Computational and Neural Mechanisms of Inference over Decision-Structure

PI: John O?Doherty, Co PI: Peter Bossaerts

ABSTRACT

In this project the Principal Investigators will determine how the brain is able to identify the relevant rules that apply to a given decision-making problem in order to effectively make decisions. In most cases, features of the decision structure are hidden variables, i.e. they can be inferred only through discrete observations of outcome variables (such as reward feedback). Understanding how inferences over decision-structure are performed in a noisy and partially observable environment is therefore a fundamental yet almost unaddressed issue in the computational neurobiology of human decision-making. Here, we conceive of inferences over decision problems as a form of hierarchical inference in which the higher level of the hierarchy represents probabilistic beliefs over which decision structure is currently in place, while the lower level of the hierarchy encodes beliefs over which actions are currently rewarded within a specific decision structure. We will compare and contrast a variety of computational models deploying different strategies to solve this problem. We will combine these models with behavioral and functional magnetic resonance imaging (fMRI) data from human participants in order to address whether dynamic signals are present in the brain pertaining to the implementation of such hierarchical models, and whether different brain regions are involved in performing inference at different levels of the hierarchy. This project could potentially lead to a new understanding of the contribution of the prefrontal cortex and other brain regions in decision-making. This project will also provide insight into the neural implementation of a fundamental missing part of the picture concerning the neurobiology of human decision-making: decision-structure inference.

In terms of broader impacts, this research could provide fundamental new insights into understanding situations where human learning or decision-making fails or breaks down. Sometimes poor learning or decision-making may be due to a failure to infer the correct rules governing a decision-problem rather than a difficulty in learning or deciding per se. Such insights will not only impact on academic fields studying decision making but could also be used to develop novel methods to help individuals and organizations make better decisions (by focusing on improving inference over structure). The findings could provide relevant data for the development of artificial agents capable of autonomous, flexible and adaptive decisions. The proposal also has high potential clinical relevance: disorders with delusional beliefs such as schizophrenia and borderline personality disorder might involve in part a difficulty in performing inference over decision structure so as to rule out inappropriate (decision) structures in lieu of more appropriate ones. The present research might yield novel tools to study this question in clinical populations. Furthermore, there are substantial impacts on teaching and training. The PI and co-PI teach courses at undergraduate and graduate level and involve undergraduate researchers directly in their research programs. The work proposed here could lead to the development of new software to enable the analysis of brain imaging data using computational models. A companion project is being funded by the German Ministry of Education and Research (BMBF).

? DESCRIPTION (provided by applicant): The study of the psychological, computational and neurobiological basis of Pavlovian conditioning is one of the longest standing research questions in psychology and neuroscience. In spite of the ubiquity and the importance of this form of learning, the computational mechanisms underlying the learning and expression of Pavlovian associations' remains poorly understood. Here, we investigate whether or not there exists two distinct forms of Pavlovian conditioning, a model-based form in which the expression of conditioned responses to a conditioned stimulus is sensitive to the incentive value of the associated unconditioned stimulus (US), and another model-free form in which conditioned responses elicited by a conditioned stimulus are insensitive to the current US value. The distinction between model-based and model-free reinforcement-learning mechanisms has received strong empirical support in the domain of instrumental conditioning, but little is known about whether or not a similar dichotomy exists in Pavlovian conditioning. Understanding the nature of the encoding of Pavlovian associations in the brain is important because of the critical role that learned Pavlovian associations might play in the maintenance of addiction, in which cues linked to drug outcomes might promote or invigorate responding for drugs, even if those drugs are no longer deemed valuable/desirable to the individual. In the present application we address this goal by performing both functional magnetic resonance imaging (fMRI) and single-unit recordings in humans while they undergo sequential Pavlovian conditioning with appetitive outcomes. We will use a number of different cutting-edge experimental and analytical techniques, including computational based analyses, multivariate pattern classification and high-resolution fMRI. We will test for the existence of these different representations in a number of distinct structures in the brain including the amygdala, orbitofrontal cortex, ventral striatum and dopaminergic midbrain. Because we will be using high- resolution fMRI, we will have the capacity to resolve the contribution of distinct sub-regions within these brain structures to model-based and model-free Pavlovian learning, including the basolateral versus centromedial amygdala, the human homologue of the core versus shell of the accumbens, different sectors of orbitofrontal cortex, and dorsal versus ventral parts of the substantial nigra and ventral tegmental area. To complement the fMRI studies, we will record from neurons primarily in the amygdala and orbitofrontal cortex in human neurosurgery patients while they while they perform one of the main tasks used in the fMRI studies, thereby enabling us to gain insight into the relationship between the observed fMRI signals and underlying neuronal activity in at least two of our key regions of interest. By combining across these different techniques and methodologies, we will be able to address the question of whether or not model-based and model-free forms of Pavlovian conditioning are implemented in parallel in the brain, and begin to gain insight into the specific contributions of different brain regions towards these two very distinct forms of learning.

Neuronal Substrates of Hemodynamic Signals in the Prefrontal Cortex    PIs: Dr. John P. O'Doherty and Dr. Doris Tsao  Institution: California Institute of Technology PROJECT SUMMARY fMRI is the dominant technique for probing human prefrontal cortex functions in cognition, learning and decision-making. This work is predicated on the assumption that fMRI activation relates in a principled manner to the underlying neuronal activity in a given area of prefrontal cortex. Yet, virtually nothing is known about how fMRI activations relate to the underlying neural computations within the prefrontal cortex. The absence of knowledge in this domain is in contrast to burgeoning work on the relationship between measured fMRI signals and neural responses in visual areas of the brain, illuminating for instance how neuronal responses in face responsive areas give rise to fMRI activations in the temporal lobes. Compared to visual cortical areas, neurons in prefrontal cortex have more sparse, heterogeneous, and functionally distributed response characteristics, thereby rendering the relationship between neuronal and fMRI responses more enigmatic. The overarching goal of this proposal is to elucidate the relationship between neuronal computations and fMRI responses in the same areas of the prefrontal cortex. To achieve this goal we will measure fMRI activity to identify the locus of activations in prefrontal cortex while separately recording neuronal activity using a multi- electrode recording system whose placement is guided by those fMRI activations. We will also probe the neurophysiological basis of functional connectivity typically found between regions of prefrontal cortex in human fMRI studies, by recording simultaneously from multiple regions identified as being functionally connected through our fMRI measurements. We will first address these questions in macaque monkeys, and then extend our findings directly to humans, scanning healthy human participants with fMRI, and making use of a rare opportunity to obtain both intracranial electrophysiological signals and fMRI scans from the prefrontal cortex in a group of human patients undergoing evaluation for surgical treatment of epilepsy. For behavioral tasks we will draw from the domain of value-based decision-making, because (a) such tasks involve multiple regions of prefrontal cortex in both monkey electrophysiology and human fMRI studies, (b) we can use virtually identical tasks with well constrained behavior in both species, and (c) most importantly, stark discrepancies exist between what is currently known about the response properties of single neurons in the prefrontal cortex of monkeys and activations measured with fMRI during decision-making tasks in humans, which are ripe for resolving with our proposed approach. By combining across these different techniques and methodologies in both humans and monkeys, we will be able to address the question of which aspects of underlying neuronal responses gives rise to the fMRI signal in prefrontal cortex. The work will provide an essential bridge between fMRI and finer-scaled electrophysiologically-based methods for studying high order cognitive function.

? DESCRIPTION (provided by applicant): The study of the psychological, computational and neurobiological basis of Pavlovian conditioning is one of the longest standing research questions in psychology and neuroscience. In spite of the ubiquity and the importance of this form of learning, the computational mechanisms underlying the learning and expression of Pavlovian associations' remains poorly understood. Here, we investigate whether or not there exists two distinct forms of Pavlovian conditioning, a model-based form in which the expression of conditioned responses to a conditioned stimulus is sensitive to the incentive value of the associated unconditioned stimulus (US), and another model-free form in which conditioned responses elicited by a conditioned stimulus are insensitive to the current US value. The distinction between model-based and model-free reinforcement-learning mechanisms has received strong empirical support in the domain of instrumental conditioning, but little is known about whether or not a similar dichotomy exists in Pavlovian conditioning. Understanding the nature of the encoding of Pavlovian associations in the brain is important because of the critical role that learned Pavlovian associations might play in the maintenance of addiction, in which cues linked to drug outcomes might promote or invigorate responding for drugs, even if those drugs are no longer deemed valuable/desirable to the individual. In the present application we address this goal by performing both functional magnetic resonance imaging (fMRI) and single-unit recordings in humans while they undergo sequential Pavlovian conditioning with appetitive outcomes. We will use a number of different cutting-edge experimental and analytical techniques, including computational based analyses, multivariate pattern classification and high-resolution fMRI. We will test for the existence of these different representations in a number of distinct structures in the brain including the amygdala, orbitofrontal cortex, ventral striatum and dopaminergic midbrain. Because we will be using high- resolution fMRI, we will have the capacity to resolve the contribution of distinct sub-regions within these brain structures to model-based and model-free Pavlovian learning, including the basolateral versus centromedial amygdala, the human homologue of the core versus shell of the accumbens, different sectors of orbitofrontal cortex, and dorsal versus ventral parts of the substantial nigra and ventral tegmental area. To complement the fMRI studies, we will record from neurons primarily in the amygdala and orbitofrontal cortex in human neurosurgery patients while they while they perform one of the main tasks used in the fMRI studies, thereby enabling us to gain insight into the relationship between the observed fMRI signals and underlying neuronal activity in at least two of our key regions of interest. By combining across these different techniques and methodologies, we will be able to address the question of whether or not model-based and model-free forms of Pavlovian conditioning are implemented in parallel in the brain, and begin to gain insight into the specific contributions of different brain regions towards these two very distinct forms of learning.

This NSF Major Research Instrumentation (MRI) Award will enable a three-year grant to purchase a major upgrade to the magnetic resonance imaging scanner used for studying the function and structure of the human brain by neuroscience researchers at the California Institute of Technology and their national and international collaborators. The award will support the upgrade of the existing Siemens Tim Trio 3T scanner at the Caltech Brain Imaging Center to the latest Siemens Prisma platform. The upgraded scanner will provide clearer and more detailed images of the human brain. Such an improvement in imaging capabilities will enable Caltech researchers to address fundamental problems such as how the brain learns from experience, how the brain makes decisions and how brains support the ability to learn from and interact with other people in social contexts. This new equipment will ultimately help Caltech researchers obtain a better understanding of how the brain works, how it is wired up, and how it may dysfunction in disease. That knowledge, in turn, will contribute to efforts to build artificially intelligent systems. The grant will also enable students and post-docs to obtain experience in using state-of-the-art brain imaging equipment, through classes taught at Caltech that offer hands-on-experience as well as through the participation of trainees in research projects that utilize the equipment. Taken together, the cutting-edge science enabled by the new equipment, and the training of the next generation of young scientists on it, will contribute substantially to cognitive, decision and social neuroscience at Caltech, in the US and worldwide.

To advance understanding about how the brain supports the capacity of humans to learn, make decisions and mediate social interactions it will be necessary to make progress in three distinct domains. First, there is a need to develop a much more detailed circuit-level understanding of the neural mechanisms underlying these various computational processes by resolving the functional properties of discrete neuroanatomical sub-divisions within each of the relevant brain areas of interest such as the amygdala, ventromedial prefrontal cortex, striatum and midbrain. Second, it is necessary to address how the various sub-processes that are implemented in these distinct sub-systems are ultimately integrated together at the systems level to drive complex behavior. Third, it will be important to characterize how the various computations and neural implementations differ across time, tasks and individuals. The Siemens Prisma scanner provides technical capabilities that are uniquely suited to advance progress in each of these three domains at the California Institute of Technology. The new platform will offer significant improvements in the quality of high resolution fMRI scans obtained from brain structures of interest, by minimizing dropout and geometric distortion, and by increasing signal-to-noise. These capabilities will also enhance the stability of the images obtained and hence improve test-retest reliability, while the improved gradient set will offer major gains in the quality of diffusion weighted imaging, and of functional connectivity data.

Project 1. Project Summary. This Project 1, directed by John O'Doherty, is a renewal of Project 2 in our current Conte Center. It aims to continue our investigation of how we can learn to make decisions by observing the choices of another person. This ability, observational learning, is present behaviorally in several species and likely constitutes the main mechanism for the acquisition of social decision-making skills in humans. However, compared to the systems we now know to mediate direct learning through personal experience, surprisingly little is known about the systems that mediate observational learning. What systems are there? How are they differentially employed depending on the context? How might this vary across individuals? We will address these questions across four Aims that test the engagement of three postulated neural systems for observational learning. Of specific interest is a system that relies on social inference, the focus theme of this Conte Center. This system, which is thought to recruit sectors of medial prefrontal cortex and the temporoparietal junction, mediates a computationally more powerful and flexible form of observational learning that requires imputing hidden states to people to explain their actions: their values, goals, and beliefs. We hypothesize, and will test, that this is the same social inference system that is engaged in standard social neuroscience tasks, such as the ?why/how? task administered to all participants under Core 2 and investigated explicitly in Project 2. This Project 1 also links to an Aim that is described under Project 4, where we will investigate the single-unit correlates of observational learning. The strong links between this Project and several others are reflected in its personnel, which include PIs from other Projects (Mobbs, Andersen, Rutishauser, Hutcherson) and post- docs shared with other Projects. The primary approach of Project 1 uses computational fMRI, which designs fMRI tasks such that regional brain activation can be fit to the parameters in a model of the observational learning process. It will test 50 healthy participants in each of 6 experiments, recruited through Cores 2 and 3, and shared in part with the participants of Projects 2 and 3. Its Aims will test how attention to specific features of social stimuli engages different systems, how social context matters (e.g., if we are observing a human or a computer), how the reliability of the different systems may influence arbitration amongst them, and to what extent there are individual differences that correlate with psychological assessment scores from Core 3, or results from experiments that shared overlapping subjects in other Projects.

Determining the explanatory utility of computational reinforcement-learning theories of goal-directed and habitual control at behavioral and neural levels PI: Dr. John P. O?Doherty Institution: California Institute of Technology PROJECT SUMMARY Accumulating evidence supports the existence of two distinct systems for guiding action-selection in the brain: a goal-directed system in which actions are selected with reference to the current incentive value of the associated goal or outcome, and a habitual system in which actions are selected reflexively, based solely on their history of past reinforcement. A computational account for these two systems has been formulated in terms of two distinct variants of computational reinforcement-learning (RL) theory: model-based (MB) vs model-free (MF) RL. Yet, empirical evidence in support of the proposed correspondence between the psychological (RDoC level) and computational RL accounts are sparse. Here we aim to comprehensively address whether the RDoC level constructs of goal-directed and habitual control can be effectively described by the computational framework of model-based and model-free RL in humans at both behavioral and neural levels. We plan to administer two distinct behavioral tasks designed to discriminate goal-directed from habitual control and MB from MF control to a large cohort of healthy participants (n=200) and an undifferentiated cohort of psychiatric patients (n=100). Our participants will perform these tasks while being scanned with fMRI, in addition to undergoing resting-state fMRI, and diffusion weighted imaging. We will also measure behavioral traits and states relevant to psychopathology in the same individuals. We will leverage individual differences across our behavioral, computational and neural measures in order to determine the extent to which the psychological constructs and computational accounts are best viewed as being one and the same, or whether by contrast they diverge in theoretically important ways. Should we detect clear differences between the psychological (RDoC) constructs and computational descriptions on any of the levels of analysis we utilize, this will motivate an iterative refinement of the computational framework to better approximate the psychological (RDoC) level constructs, to be accomplished in parallel to the experimental aims. The distinction between goals and habits and their proposed computational bases are arguably one of the most influential research topics in computational psychiatry to date, given the hypothesized relevance of these constructs as a means of capturing various forms of psychiatric dysfunction. Thus, a better understanding of the nature of the relationship between these constructs, coupled with a process of active refinement of the computational theory to achieve a much closer correspondence to the psychological constructs, is going to be critical for progress in this domain.

TOWARD A HIGH DIMENSIONAL COMPUTATIONAL DESCRIPTION OF VARIATION IN HUMAN DECISION-MAKING ACROSS PSYCHIATRIC AND NON-PSYCHIATRIC POPULATIONS PI: Dr. John P. O'Doherty Institution: California Institute of Technology PROJECT SUMMARY Computational psychiatry (CP) promises to gain deeper explanatory insight into psychiatric disorders through the application of formal computational models to task-related behavioral data and brain measures. However, research in CP to date has mostly involved a narrow unidimensional focus, utilizing either a relatively limited set of computational constructs such as simple model-free (MF) reinforcement learning (RL), and/or restricted to studying a specific task, a specific disease, or even a particular model parameter. For CP to reach its potential, we need to broaden the field's scope. To achieve this, it is necessary to develop an integrated theory and formal framework supported by a task battery that will enable the quantification of individual differences across a range of computational mechanisms pertinent to the diagnosis and treatment of clinical disorders. The objective of the current proposal is to implement the initial groundwork needed to build and test a computational framework and task battery that can facilitate a multi-dimensional computational description of individual variability in parameters relevant for characterizing psychiatric dysfunction. We have constructed a computational assessment battery (CAB) consisting of four distinct yet inter- related tasks that move beyond simple RL to probe various aspects of learning, cognition, and decision-making. First, we assess learning about losses as well as rewards. Secondly, we measure model-based (MB) alongside simple MF learning and decision making, and the arbitration allocating control to either strategy. Thirdly, we examine strategies for solving the exploration/exploitation dilemma, in which individuals have to decide whether to exploit an option known to yield reward or explore an option whose outcomes are unknown. Finally, we assess social-learning, in which an individual can either infer the goals of another individual or simply imitate that agent's behavior. We propose to build an integrated computational model that can capture relevant computations being implemented in each of the tasks in our battery, alongside a hierarchical model-fitting and parameter estimation framework to enable us to retrieve reliable parameter estimates for each computational variable of interest. We will leverage common computational mechanisms engaged across our task battery to improve estimability and generalizability. We will then acquire behavioral data in a large on-line (n=1000), and modest (n=100) in-lab sample on the CAB to establish the internal validity and test/re-test reliability of our computational model and parameter estimates. Finally, we will explore the relationship between model-estimated parameter estimates from behavior on our task battery and variability in self-reported traits and states relevant for psychiatric disease in our healthy samples, as well as in a pilot sample (n=60) of patients recruited from the UCLA psychiatry outpatient clinics. These efforts will form the basis of a Computational Psychiatry Toolbox for behavioral testing, model-fitting and parameter estimation that will eventually be released as a resource to the research community.