James Haxby - US grants

Appropriate and effective social interaction requires ready access to representations of the personal traits, intentions, goals, opinions, and mental states of others. With funding from the National Science Foundation, James V. Haxby is investigating the functional organization of neural systems that extract such person knowledge from the perception of faces and gestures. The proposed experiments use functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and other measures to investigate the roles played by neural systems for face perception, action perception and imitation (mirror neuron, MN), and theory of mind (ToM) in obtaining social information from faces. This information can be transient mental states, such as direction of attention, emotional state, and level of interest, or more enduring attributes, such as personal traits, goals, attitudes, and beliefs. Perception of facial and social gesture, on the one hand, and recognition of familiar faces, on the other, are hypothesized to be mediated by distinct parts of the neural systems for face and action perception. By contrast, a common neural system for the representation of person knowledge is hypothesized to mediate the representations of transient mental states and enduring personal attributes of others that are activated during face perception. The system for these representations is hypothesized to be the same system that has been associated with ToM, the ability to represent the mental states of others. The first objective of the experiments is to characterize the functional organization of the mirror neuron system, its role in the perception of facial expressions, social gestures, and eye gaze, and its role in the activation and updating of representations of transient mental states in the ToM system. The hypothesis is being tested that perception of facial movements and gestures activate mirror neuron networks (superior temporal sulcus, intraparietal sulcus, and premotor regions, namely Broca's area and the frontal eye fields)' that both perception and imitation evoke this activity; and that movements that convey socially-relevant information also will evoke activity in the ToM system. The second objective is to investigate which of the regions that are activated during the recognition of familiar faces are associated with the spontaneous retrieval of representations of the enduring personal attributes associated with familiar individuals. The hypothesis is being tested that this process is mediated by cortical areas in the ToM system (anterior paracingulate cortex, posterior superior temporal sulcus), and these areas can be distinguished from areas associated with emotional responses (amygdala) and the retrieval of autobiographical episodic memories (posterior cingulate/precuneus, anterior temporal cortex). Automaticity of retrieval of person knowledge associated with familiar faces by manipulating awareness of the faces will also be investigated.

Face perception is a highly developed and efficient skill that plays a central role in social communication. In particular, face perception may play a more critical role than language in conveying information about the intentions, attitudes, emotions, level of interest, and other transient mental states of others. Better understanding of the functional architecture of the human neural systems that mediate this skill can help to inform computational approaches to face perception, with possible applications to improved human-computer interfaces. Such understanding can also help in the development of diagnostic procedures and therapies for psychiatric disorders characterized by impaired social communication and face perception, such as social phobia, autism, and schizophrenia. Undergraduate and graduate students at Princeton University will be involved in the research.

DESCRIPTION (provided by applicant): fMRI experiments produce large, numerically rich, but noisy data sets that pose a challenge for extracting the signal variance and establishing the correspondence between that signal and cognitive variables. Conventional analysis has reduced the dimensionality of fMRI data by searching for clusters of voxels that show similar responses to experimental manipulations and averaging the signal within those clusters. We have introduced a new approach to fMRI data analysis, "multi-voxel pattern analysis", that examines higher spatial frequency patterns of activity - the voxel-by-voxel variation of response within a region - and have shown that this method greatly increases the sensitivity of fMRI (Haxby et al. 2001;Hanson et al. 2004;OToole et al. 2004;Polyn et al. 2004). In the proposed investigations, we will develop new methods for analysis of spatially-distributed patterns of neural activity in relation to two specific problems in fMRI data analysis: 1. accounting for inter-individual variation in functional neuroanatomy, and 2. the relation between spatially-distributed neural population responses and cognitive representations. This work will involve the efforts of a multidisciplinary team consisting of cognitive neuroscientists, applied mathematicians, and signal- processing engineers. We propose the development of analytic methods for aligning the functional neuroanatomy of individual brains based on the patterns of neural activity that are elicited by a broad spectrum of cognitive activities. We predict that these methods will enhance the sensitivity of group statistical tests of fMRI data, will allow the investigation of the inter-individual consistency of higher spatial frequency topographic representations, and will provide explicit measures of inter-individual variation in the location, organization, and spatial extent of functional maps, with potential applications for studies of clinical conditions. We propose, further, to develop methods for detecting and analyzing distributed patterns of neural activity that make use of prior knowledge about the structure of the cognitive representations that are associated with those neural activities. We predict that these methods will increase the sensitivity of multi-voxel pattern analysis and will allow the investigation of how cognitive information is represented in topographically-organized, spatially-distributed patterns of neural activity.

Methods known as 'multivariate pattern' (MVP) analysis can be used to decode the information patterns in brain activity obtained using functional magnetic resonance imaging (fMRI). However, a new decoding model has to be built for each brain, because two brains (and the representational spaces they employ) are difficult to align at a fine spatial scale. As a consequence, we do not yet know if different brains use the same codes or idiosyncratic codes to represent the same things. With funding from the National Science Foundation, Drs. James V. Haxby of Dartmouth College, and Peter J. Ramadge of Princeton University, in collaboration with Michael Hanke of the University of Magdeburg (Germany), are developing new methods to discover a coding scheme that works accurately across different brains. The methods being developed align brain activity across brains by projecting individual brain data into a common, high-dimensional space. This approach allows the researchers to build models of brain representational spaces for different cortical areas that are valid both across brains and across a wide range of stimuli and cognitive states. The researchers are developing two algorithms. One is referred to as 'hyperalignment' and the other as 'functional connectivity hyperalignment.' Hyperalignment rotates the voxel spaces (i.e., the smallest units in a brain image) of individual brains into a single high-dimensional space, in which each dimension is a profile of differential responses to stimuli, that is common across brains. Functional connectivity hyperalignment aligns voxel spaces based on the functional connectivity profile (i.e., relationships among activated brain areas) for each cortical location. Functional connectivity profiles allow for models of areas that do not respond to external stimuli in a consistent manner, for example, those areas in the so-called 'default-intrinsic system' that plays a central role in social cognition. The investigators are an interdisciplinary partnership - cognitive neuroscientists and signal-processing engineers - who have been working together successfully for several years.

Developing the computational methods to build common models of representational spaces will augment the power of brain activity decoding techniques, making it possible to investigate how finer, more detailed information is embedded in brain activity patterns, and to read out that information from functional brain imaging data. The proposed methods also will allow extension of brain decoding to the neural codes that underlie social cognition, that is, the representation of knowledge about the personal traits and mental states of others. These models also will allow investigation of how neural coding is altered within brain regions that are affected by experience, by development, and by psychopathology.

This project is jointly funded by Collaborative Research in Computational Neuroscience and the Office of International Science and Engineering. A companion project is being funded by the German Ministry of Education and Research (BMBF).

The human brain is perhaps the most complex known object and is the vessel that contains our thoughts, experiences, knowledge, and, collectively, our culture. Although brains have similar anatomical components, differences in size and shape and in fine structure make it difficult to discern how different brains can contain similar thoughts and knowledge that can be shared and communicated. A major challenge for brain science is to build a common model of the functional architecture of the brain that captures these similarities that are shared across brains at a fine scale. The research in this project is aimed at developing a computational basis for such a common model. The model is based on measurement of patterns of brain activity using functional magnetic resonance imaging (fMRI) while participants engage in everyday cognitive activities like watching a movie, listening to a story, or free-ranging thought while at rest. The model aligns the functional architecture of the brain at multiple scales, from fine to coarse, and captures far more shared structure than is possible with other methods that are based on alignment of brain anatomy. This model will provide infrastructure that can be used by scientists who image the brain in order to study a wide range of brain functions, from perception to social interaction, emotion, and decision making, allowing them to describe the mechanisms underlying these functions in a format that can be communicated across laboratories with a level of detail and precision that will accelerate discovery and application.

Alignment of brain imaging data has relied on anatomical features that have a variable correspondence to the underlying functional architecture. Moreover, such alignment methods do not capture the fine structure of brain activity patterns that can be decoded using modern pattern analytic methods. The research in this project is based on aligning representational spaces across brains, rather than anatomical topographies, and will identify the boundaries between patches of cortex with distinct representational spaces. This innovation affords greatly superior alignment of functional architecture across brains and the development of a common model of the human brain. The research will develop computational algorithms for transforming the idiosyncratic organization of individual brains into the common representational spaces and for fine-tuning the description of individual brains by projecting, or shrink-wrapping, the common model based on a large number of individuals onto that individual brain. The development of these computational algorithms and the model will be integral to the training of graduate students and postdoctoral fellows. They will be made available as free and open-source software, with large shared data sets, to be shared and used freely by brain imaging scientists around the world, providing essential research infrastructure to maximize the impact and benefit of this research project.

Project Summary How the self is experienced is central to healthy emotional functioning as well as many disturbances in psychological functioning. This competing renewal uses structural, functional, and resting-state neuroimaging, coupled with passive smartphone sensing technology and ecological momentary assessments, to examine the affective components of self. Understanding the factors that contribute to changes in the affective aspects of self that result from environmental stressors has the potential to provide important insights into the development of mental disorders and help identify individuals who might be in greatest need of early intervention or treatment. Research findings during the prior two award periods (R01 MH059282) revealed several key brain regions involved in processing information related to self. Moreover, we discovered that structural and functional connectivity between these regions and other brain regions known to be involved in emotional processes are associated with measures of self-affect. The overarching goal of this research is to examine how brain connectivity and activity is related to change in subjective distress and associated functional impairment. An exciting aspect of the proposed work is that we will take advantage of the university setting to follow a large cohort of participants over their four years of college to assess how changes in self- affect are predicted by relevant brain networks as well as how those networks change over time. Tasks assessing self-affect will be performed during scanning. Given that approximately 30% of participants are likely to develop a significant subjective distress, one goal is to examine whether there are biomarkers that predict these outcomes. Additional scanning studies will induce interpersonal distress to examine the temporary inductions of affect on task performance. This project will use recently developed applications of network analysis to assess resting state connectivity in brain circuitry and its relation to self-affect and health- relevant outcomes. The guiding hypothesis of this research is that individual differences in the integrity of these networks can predict individual differences in vulnerability to stress and their relation to self-affect. The specific aims of the study are: (1). Characterize neural networks that give rise to self-affect using diffusion tensor imaging, resting state functional connectivity, and task-related functional imaging. In addition, multivariate pattern analysis and representation similarity analysis will be used to classify participants as having high or low self-affect (e.g., self-esteem, depression, anxiety); (2). Examine how changes in self-affect that occur over time are reflected by changes within relevant brain networks and are predicted by baseline network connectivity; and (3). Examine how induced interpersonal distress impacts self-affect and related functional connectivity across networks. Understanding the factors that contribute to changes in self-affect that result from environmental stressors has the potential to provide important insights into the development of mental disorders and help identify individuals who might be in greatest need of early intervention or treatment.

This research project will study how brain systems for face perception differ across individuals. Face perception plays a central role in interactions among people. The ability to recognize faces and interpret expressions can vary greatly in healthy adults. Efficient face perception develops slowly through childhood and into early adulthood. Face perception is much more efficient for familiar individuals with whom we interact frequently. We will use new, state-of-the-art methods for modeling the brain's system for face perception. The model has interacting processing pathways. Each pathway serves a different function. These include recognition of identity, interpretation of expression, and activation of social knowledge. We study individual differences using a new approach, called hyperalignment. Hyperalignment allows us to see how information is encoded in fine-scale brain patterns. These studies can make it possible to address questions about the effects of development, education, culture, and clinical disorder on brain organization.

The project will investigate individual variation in the human cortical functional architecture for face perception that leverages our previous work on multivariate models of information in the distributed neural system for face perception and our work creating hyperalignment to build high-dimensional common models of information spaces in cortex. Our approach discovers shared basis functions for information that is encoded in fine-scale cortical topographies, affording reliable measurement of individual differences in the representation of this detailed information. We will investigate individual variation in cortical systems for face perception as a function of cognitive ability, development, and learning, and build the common model of cortical information spaces using fMRI data collected during viewing of naturalistic movies and in the resting state. We will use response hyperalignment and connectivity hyperalignment to derive a common model of the face perception system with shared basis functions for fine-scale variation in response tuning and functional connectivity. By modeling shared neural representation at a fine scale, measures of deviations from shared representation are more sensitive to the inter-individual variation that underlies differences in cognitive function. Our methods have the potential to provide a firmer and more nuanced basis for addressing questions about the effects of development, education, culture, and clinical disorder on brain organization.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.