Nancy Kanwisher - US grants

Repetition blindness is the failure to detect or recall second instances of repeated words presented in rapid serial visual presentation (RSVP). Repetition blindness is robust and has been shown to arise at rapid presentation rates because the second occurrence of a repeated words is frequently recognized as a type but not episodically individuated as a distinct token from the first occurrence. The experiments proposed here address three main questions about REpetition blindness (RB): i) At what locus is processing of the second occurrence of a repeated word blocked?, ii) Does the same processing mechanism underlie both token individuation and feature conjunction?, and iii) Is RB a consequence of the way in which recognized items become stably encoded in STM? These questions will be pursued using both RSVP and simultaneous displays of feature arrays with a variety of tasks such as free recall, repetition detection, numerosity judgements, and partial report of stimulus lists and arrays. Many of the experiments will manipulate repeatedness of target items, predicting in general that items will be harder to report if the stimulus contains another item of the same type, than if it does not (this is the hallmark of RB). Repetition blindness merits study not simply as a curiosity and a counterintuitive phenomenon, but because it demonstrates an important dissociation between the processing of visual types and of visual tokens. Many important but currently unconnected areas of vision research, which deal with the organization of incoming categorical information into episodic frameworks, can be recast in terms of the integration of type and token information. This proposal attempts to show that i) the distinction between type and token processing provides a common conceptual framework in which to consider diverse visual phenomena, and ii) repetition blindness provides an ideal methodologic tool with which to study the interrelations among these phenomena in detail.

How do we construct organized and meaningful representations of the visual environment from the fragmentary featural information provided by early vision? Recognition--determining the types of objects present in a visual scene--is only half the problem. We also need to be able to establish a separate representation for "object token") for each instance of an object, specifying that object's position (or trajectory) in space and time. My work on repetition blindness (RB) has demonstrated a fundamental dissociation in human vision between the recognition of visual types and the individuation of visual object tokens. In RB, although the repeated stimulus item is recognized, it is not perceived consciously because it does not get assigned to a new object token. The experiments proposed here investigate the mechanisms of token individuation (Part I) explore what visual processes require token individuation (Part II), and examine the relationship between repetition blindness and other attentional phenomena (Part III). Experiments 1 - 3 investigate why repetition blindness is robust at stimulus presentation rates of 7 or more items/second but nonexistent at 4/second, by exploring the dependence of RB on critical word duration, total memory load, and other factors. Is RB due to the lack of a token altogether, or the lack of a new type-to-token link (Experiment 4)? Is RB reduced or eliminated when the two occurrences are perceived as part of the same object token - because they are either connected behind an occluder (Experiment 5) or connected by apparent motion (Experiment 6)? The answers to these questions will be necessary before a full account of the mechanisms underlying RB can be offered. Part II uses RB to ask what visual processes require object tokens, and explores the nature of the visual "type" categories underlying repetition blindness. Is token individuation necessary for semantic priming (Experiment 7), scene recognition (Experimetn 9), and feature conjunction (Experiment 10)? Does RB require a preexisting category at all, or will it happen for novel shapes (Experiment 11) or nonwords (Experiment 12)? Part III explores the possible connections between repetition blindness and other attentional phenomena. Experiment 14 asks whether a single very general novelty bias in perception might underlie both RB and novel pop-out. Experiment 15 tests whether it is only new types, or also new object tokens, which automatically pull attention. Experiment 16 asks whether attentionally "blinked" items will cause negative priming (but not RB).

DESCRIPTION (Applicant's Abstract): The goal of the present research is to use Functional Magnetic Resonance Imaging (fMRI) of the brain to discover the functional components of visual recognition. This application takes as its starting point the recent discovery of an area in extrastriate cortex which is specifically involved in the analysis of visual object shape. The experiments described here will test subjects in an MRI scanner on a wide variety of visual stimuli and tasks in order to 1) zero in on the specific processes that take place in this cortical area and 2) discover new brain areas specialized for different functional components of visual recognition. Part I will test whether visual recognition of different classes of stimuli (objects, faces, words and biological-motion displays) engage the same or different cortical areas. Two aspects of the experimental design will permit a more precise answer to this question than has previously been possible. First, stimuli will be constructed which control for low-level feature components of complex images. Second, detailed within-subject analyses will allow the investigators to dissociate different functional areas even if the anatomical loci of these areas varies considerably across subjects. In Part II, a more fine-grained analysis of the different components of visual shape analysis will be carried out in an effort to distinguish between cortical areas involved in figure-ground segregation, part decomposition, depth interpretation, and other components of shape processing. The investigators will also look for cortical areas that respond differently to familiar and unfamiliar visual stimuli in order to determine whether visual memories and memory-matching processes are separable from bottom-up stages of visual processing. In order to explore the role of attention in visual object recognition, in Part III the investigators will manipulate the subjects' attention with a variety of tasks to ask whether the shape of an object is analyzed even when attention is focused on 1) the color of the same object, 2) another object in the visual field, or 3) a difficult perceptual task in another sensory modality. Using fMRI imaging in conjunction with carefully-controlled stimuli and tasks to characterize and localize specific visual computations is a new way to exploit the modular structure of the brain to discover the functional components of the mind. The present approach may provide the crucial missing link between the considerable knowledge of the neural basis of vision and the understanding that has been gained from computational and cognitive approaches.

All navigating organisms must answer the fundamental question, where am I now? FMRI research in my lab over the last 8 months has identified a region of cortex we call the parahippocampal place area (PPA), which appears to play a central role in solving that problem. We have functionally localized the PPA in 28 out of 30 subjects tested and have shown that the PPA responds in a highly selective fashion to visual stimuli depicting places, responding robustly when subjects view photographs of indoor and outdoor scenes, yet only very weakly to photographs of common objects and faces. The goal of the proposed research is to specify the precise function of the PPA associated neural structures (e.g., the hippocampus, perirhinal and entohinal cortex, and parietal regions) in place perception, navigation, and spatial cognition. The experiments in Part I lay the groundwork for future studies by characterizing the anatomical and functional properties of the PPA and associated structures in detail. The experiments in Part II are designed to precisely characterize the kinds of stimuli that drive the PPA. Having clearly defined the categories of stimuli that activate the PPA, Part III will determine the nature of the processing the PPA and associated neural structures carry out on those stimuli. Specifically, these experiments test the involvement of the PPA and other regions in I) the perceptual analysis of the stimulus (the Perception Hypothesis), ii) the comparing of that perceptual information to stored representations of the appearances of places (the Recognition Hypothesis), iii) the encoding of theat perceptual information into memory (the Memory Encoding Hypothesis), and/or iv) determining how to get from the current location to some other known location in the cognitive map, i.e. route planning (the Route-Planning Hypothesis)

To understand mathematical cognition both as it develops in the young child and as it is taught in school, one must understand the cognitive systems from which it is constructed and the processes by which those systems are coordinated to produce new concepts and skills. Based on previous research, we hypothesize that elementary school mathematics builds on three representational systems: a system for representing exact small numerosities, a system for representing approximate large numerosities, and natural language with its system of number words and other quantifiers. The proposed research investigates each of these building block systems and their interactions through experiments on human infants, non-human primates, preschool children learning counting, elementary school children learning arithmetic and fractions, and adults.
To study the building block cognitive systems directly, experiments investigate spontaneous number representations in human infants and in untrained adult monkeys, using in each population the same three converging behavioral measures: looking time to arrays of different numerosities and to addition or subtraction events (building on the finding that both infants and monkeys look longer at novel arrays or unexpected events), manual search (building on the finding that the number of times that an infant or monkey will search in a container depends on the number of objects it represents within the container), and locomotor approach to containers with different numbers of attractive objects (building on the finding that infants and monkeys will approach the container with the greater number of objects). Further experiments investigate how preschool children assemble these components in learning number words and the counting routine, by using verbal and pointing tasks to assess developmental changes in children's understanding of number words and counting procedures. To uncover the neural substrates underlying mathematical cognition, both behavioral and neuroimaging experiments investigate whether and how human adults use each of the three representational systems in performing numerical comparisons and elementary arithmetic. Finally, experiments investigate number concepts and arithmetic learning in elementary school children. Training studies in which children are taught new facts or concepts and then are tested on a range of related problems will serve to investigate the subsystems involved in this learning, to probe the processes by which those subsystems are assembled to meet new educational challenges, and to explore ways of enhancing mathematics learning in elementary school.
This research promises to shed light on the teaching and learning of mathematics through coordinated, laboratory-based studies in which monkeys, infants, children and adults are given the same stimuli and often the same tasks. This coordinated effort should provide a broad portrait of the sources of mathematical thinking, from its phylogenetic and ontogenetic origins to its culmination in educated adults.

[unreadable] DESCRIPTION (provided by applicant): The training program in Visual Cognition is based on the beliefs that vision is central to the understanding of the mind, and that it can now only be studied in a way that integrates knowledge and methods from many traditional disciplines (cognitive psychology, psychophysics, neuroscience, neuropsychology, computer vision). This requires new generations of scientists whose training encompasses all these disciplines. MIT is a perfect site for such a program because of its long history of accomplishments in vision science (both research and training), the unusual composition of the department (experimental psychology, computation, and neuroscience), and its experience and infrastructure for an interdisciplinary training program contributed by the Department of Brain and Cognitive Sciences. The program has provided predoctoral students with core courses in cognitive science and neuroscience, extensive research experience, oral and written qualifying exams, lecture courses and research seminars, experience and training in undergraduate teaching, close oversight of progress, repeated oral presentations, and immersion in a peer culture of students and fellows interested in visual cognition. Now is a crucial time for the renewal of this training grant because of the great expansion in vision research we expect in our department over the next few years, resulting from the increased access to brain imaging equipment at the new MIT-MGH Martinos Center for Biomedical Imaging, and from the numerous new hires expected in the McGovern Institute for Brain Research (of which the PI is a member), several of whom are likely to work on high-level vision. [unreadable] [unreadable]

Attention and Performance Meeting
ABSTRACT

With National Science Foundation support, junior researchers will be provided the funds to participate in the XXth meeting of the International Association for the Study of Attention and Performance (IASAP). This meeting will be held from July 1-7 in Erice, Sicily, and will address the topic of Functional Brain Imaging of Visual Cognition. Functional brain imaging may have by now surpassed all other techniques in cognitive science in terms of expense, growth rate, and public visibility. But how much has this new set of techniques actually contributed to the study of human cognition? When if ever has a finding from brain imaging constrained a cognitive theory? We are sympathetic to those who have watched from the sidelines and felt underwhelmed by the contributions of brain imaging to cognitive science. Many of the early imaging studies in fact had little to offer researchers interested in cognition. However, we are encouraged by more recent results and hopeful that a new era is beginning in which functional brain imaging may realize its potential as a powerful tool for the study of cognition. It is the goal of this meeting to tackle all of these issues by addressing whether, when, and how functional brain imaging can constrain theories of human cognition. We have chosen to focus our discussions around the specific topic of visual cognition, where much of the best recent imaging work has been carried out. We have selected 65 of the top researchers in the world who either do imaging work on visual cognition, or who have expertise in a closely related field the understanding of which is critical for our advancement of brain imaging research (e.g., cognitive psychology, single-unit recording, computational modeling). At the meeting, these researchers will present the latest results from their labs and discuss the ways in which imaging can and cannot provide answers to cognitive questions. The meeting will include sessions on modularity, visual object representation, development and plasticity, visual attention, and sensorimotor integration. Every effort was made to include minorities, women, and researchers at early stages of their careers. We anticipate that several factors will guarantee that this meeting will have a broad impact not only on those attending it, but also on the field as a whole. These factors include the prestige of the IASAP, the high visibility of many of the meeting participants, and the long tradition of publishing from each Attention & Performance meeting an excellent and widely-cited volume of articles.

The XXth meeting of the International Association for the Study of Attention and Performance (IASAP) will be held from July 1-7 in Erice, Sicily, and will address the topic of Functional Brain Imaging of Visual Cognition. Functional brain imaging may have by now surpassed all other techniques in cognitive science in terms of expense, growth rate, and public visibility. But how much has this new set of techniques actually contributed to the study of human cognition? When if ever has a finding from brain imaging constrained a cognitive theory? We are sympathetic to those who have watched from the sidelines and felt underwhelmed by the contributions of brain imaging to cognitive science. Many of the early imaging studies in fact had little to offer researchers interested in cognition. However, we are encouraged by more recent results and hopeful that a new era is beginning in which functional brain imaging may realize its potential as a powerful tool for the study of cognition. It is the goal of this meeting to tackle all of these issues by addressing whether, when, and how functional brain imaging can constrain theories of human cognition. We have chosen to focus our discussions around the specific topic of visual cognition, where much of the best recent imaging work has been carried out. We have selected 65 of the top researchers in the world who either do imaging work on visual cognition, or who have expertise in a closely related field the world who either do imaging work on visual cognition, or who have expertise in a closely related field the understanding of which is critical four our advancement of brain imaging research (e.g., cognitive psychology, single-unit recording, computational modeling). At the meeting, these researchers will present the latest results from their labs and discuss the ways in which imaging can and cannot provide answers to cognitive questions. The meeting will include sessions on modularity, visual object representation, development and plasticity, visual attention, and sensorimotor integration. Every effort was made to include minorities, women, and researchers at early stages of their careers. We anticipate that several factors will guarantee that this meeting will have a broad impact not only on those attending it, but also on the field as a whole: i) the prestige of the IASAP, ii) the high visibility of many of the meeting participants, and iii) the long tradition of publishing from each Attention & Performance meeting an excellent and widely-cited volume of articles.

[unreadable] DESCRIPTION (provided by applicant): The remarkable human facility with social cognition depends on a foundational ability to reason about other people based on an understanding of their minds. The specific aims of the work proposed here are to i) test whether we have special ("domain specific") cognitive and neural mechanisms for detecting and reasoning about other minds, ii) discover the functional organization of this system (i.e., what are its main components?), and iii) characterize the processes that go on, and the representations that are extracted, in each of these components. Our experiments will use fMRI to identify regions in human temporal and frontal cortex that are involved in understanding other minds. The purpose of this work is not simply to discover the anatomical locus of these regions, but to use fMRI to identify and characterize the functional components of the system. In particular, in Part I the Domain Specificity Hypothesis will be tested, by asking whether any brain regions are more active in conditions that involve detecting or reasoning about other minds than in control conditions that do not involve other minds but that are matched for complexity, difficulty, and logical structure. We will also test whether any candidate cortical regions implicated in detecting other minds are engaged whenever the relevant stimulus information is present, independent of the task the subject is asked to carry out (the Automaticity Hypothesis). Parts II and III describe experiments investigating the functional architecture of the system identified in Part I, by testing whether this system consists of a single mechanism that is involved in all aspects of perceiving and reasoning about other minds (the Single Component Hypothesis), or whether it consists of several functionally dissociable components, each instantiated in a different cortical region (the Multiple Components Hypothesis). These experiments will also test specific hypotheses concerning the function of each component. The experiments proposed here constitute the first broad- based effort to use neuroimaging to characterize the functional architecture of one of the core components of human cognition: detecting and reasoning about other minds. Progress on the experiments outlined here will provide a solid foundation for future research exploring the recruitment of these core systems in everyday social behavior, the development of these systems in childhood, and the disruption of this system in neurological patients, autism, and psychopathology.

The aim of this SLC Catalyst activity is to develop plans for an interdisciplinary multi-institutional center devoted to translating contemporary research on perceptual learning and brain plasticity into educational and rehabilitation outcomes, with special emphasis on the needs of the 3+ million Americans with impaired vision. The Center will be composed of a partnership among teams of researchers at five leading sites for vision research: University of Minnesota, Vanderbilt University, Boston (MIT, Harvard, and Shepens Eye Research Institute), UCLA, and the Max Planck Institute for Biological Cybernetics (Tubingen, Germany) . These teams combine interdisciplinary expertise in perceptual development, impaired vision and special education, computational vision, neuroscience, and behavior.

The key idea behind the proposed center is that learning and adaptation take place within the visual and other perceptual pathways. Understanding the principles of learning at this early input stage of information processing is a critical prerequisite for understanding human capacities that rely on sensory input.

The following planning activities will be completed: 1) Identify the most important educational and rehabilitation needs related to perceptual learning and brain plasticity in reading, recognition and spatial navigation. 2) Identify the existing research barriers to solving them, and potential research solutions. 3) Hold a workshop to arrive at a consensus for a center's educational and rehabilitation priorities. 4) Incorporate these plans into a full Center proposal.

Because of its intrinsic importance to human nature, vision is a model system for researchers interested in development, neuroscience, cognition and behavior, and artificial intelligence. Vision is also a topic of primary concern to educators and rehabilitation specialists with visually impaired students or clients, and to engineers and computer scientists who design adaptive technology for visually impaired people. For the most part, these two communities -- vision researchers, and vision educators and engineers -- have had little interaction, leaving unfulfilled opportunities for translating the huge body of research findings into improved quality of life for visually impaired people. Three burgeoning areas of vision research provide the basis for bridging this gulf -- perceptual learning, brain plasticity, and machine learning. Recent advances in these three areas has overturned the prevailing view that human vision is frozen in structure following a brief early critical period. There is a sea change in thought in which the visual system (and presumably other neural systems) is beginning to be regarded as modifiable throughout the human lifespan. Just how modifiable, on what time scale, and at what level of specificity are key unresolved issues to be addressed.

[unreadable] DESCRIPTION (provided by applicant): This proposal is in response to PA-03-107 (NIH EXPLORATORY/DEVELOPMENTAL RESEARCH GRANT AWARD R21). FMRI investigations over the last decade have revealed the functional organization of human visual cortex in considerable detail, including the identification of a number of extrastriate areas, each of which responds quite selectively to a specific kind of visual information: the fusiform face area (FFA) for faces; the parahippocampal place area (PPA) for the spatial structure of scenes, the extrastriate body area (EBA) for bodies and their parts, the lateral occipital complex (LOC) for object shapes, and visual motion area MT. Almost nothing is known about the developmental origins of these extrastriate areas. The experiments described here will first behaviorally test children age 4-adult on a set of visual tasks, each designed to tap the function of a different extrastriate area, then characterize the development of these extrastriate regions directly using fMRI (in children age 7-adult), and finally test the relationship between the developmental changes observed neurally and those observed behaviorally. We will test the hypothesis that cortical specialization increases throughout childhood (as expected if the cortex is fine-tuned through experience), against the competing hypothesis that informational exchange between processing modules increases over development, thereby reducing domain-specific activation patterns in these areas. We will also test specific hypotheses about differences between areas in the time course of developmental change. Finally, we will test relationships between the developmental changes we observe behaviorally vs. neurally with longitudinal studies of individual subjects. Extrastriate cortex is an ideal test case for pediatric fMRI because BOLD signals are very large from this region of the brain, and because complex tasks are not required to activate this region. In addition to advancing the methodology of pediatric fMRI, this work is likely to provide important insights about i) the developmental origins of human extrastriate cortical areas, ii) the behavioral functions that each area subserves, iii) the nature and developmental change in visual shape representations across childhood, and iv) the domain-specificity of high-level vision and its neural substrate. [unreadable] [unreadable]

DESCRIPTION (provided by applicant): Visual object recognition is widely considered to be the central problem of high-level vision. The development of functional MRI (fMRI) has provided a powerful new approach for investigating object recognition, by enabling us to characterize the representations that underlie object recognition and to ask how they change with experience. Here we ask in Specific Aim I what information the ventral visual pathway represents about objects. We use "event-related fMRI adaptation" to test three of the main current proposals, that object representations are composed of i) representations of concave and convex contour segments (Hypothesis I), ii) intuitive object "parts" (Hypothesis II), or iii) wholes (Hypothesis III). In Specific Aim II, we ask how neural representations of objects change with experience. Here we scan people while they view novel stimuli, before and after extensive training on discriminating those stimuli, to test whether this training changes the overall response to the trained objects (Hypothesis IV), increases the sharpness of tuning of neural populations responsive to the trained objects (Hypothesis V), and/or leads to selective responses to the trained stimulus class in focal regions within the ventral visual pathway (Hypothesis VI). We will also test whether training creates representations of trained objects that are specific to the position where they were presented during training (Hypothesis VII), and finally whether training increases the coding of specific combinations of image components (Hypothesis VIII). In Specific Aim III, we ask whether functional heterogeneity in the ventral visual pathway corresponds to anatomical heterogeneity. Here we will coregister functional architecture characterized with fMRI, to post mortem histology, to test Hypothesis IX that cytoarchitectonic subdivisions exist between and within object processing regions of the ventral visual pathway. The methods developed in the first project period have opened up new opportunities for us to make rapid progress answering three of the most fundamental questions about the neural basis of visual recognition: the nature of the representations underlying object recognition, the effects of experience on those representations, and the way those representations differ across distinct regions within the ventral visual pathway.

DESCRIPTION (provided by applicant): Macular degeneration (MD)-the loss of central vision due to retinal damage-is the leading cause of visual impairment in the developed world, affecting more than 1.6 million Americans over the age of 50. Individuals with loss of central vision must learn to cope with peripheral vision only, which has low acuity and contrast sensitivity. In normal subjects, a large region of visual cortex is allocated to processing visual information at the center of gaze, and an understanding of what happens to this cortical region when its input is cut off by MD will be critical in any effort to develop better methods of vision rehabilitation. Pilot fMRI data from our lab shows a striking and previously undescribed phenomenon in which the region of visual cortex that responds to the fovea in normal subjects is strongly activated by peripheral stimuli in MD subjects, indicating functional reorganization of retinotopic cortex (FRRC) in MD. The research outlined in this proposal will use fMRI scanning of MD subjects as well normal control subjects while they view visual stimuli, in order to test the following hypotheses about MD: i) that FRRC occurs rapidly after the onset of MD and strengthens over time, such that we will find FRRC in every binocular MD subject who has central field loss, including those who have developed the disease very recently, ii) that FRRC occurs even in monocular MD, although more slowly than in binocular MD, iii) that formerly foveal cortex will respond primarily or only to stimuli presented in the "preferred retinal locus" (i.e. that part of the surviving retina that MD subjects use preferentially for tasks such as reading and face recognition), iv) that FRRC will be found only when the scotomas cover the fovea, not when the fovea is spared and the peripheral visual field is compromised, and v) that formerly foveal cortex contributes to visual performance in MD subjects, but does so for peripheral (rather than central) visual space. This research is important for three reasons: 1) It will inform research into the basic neuroscience of cortical plasticity and its relationship to behavior; 2), It will answer fundamental questions about the neural mechanisms by which MD subjects cope with loss of central vision; 3) It will help guide the search for better rehabilitation strategies for people with MD.

This project will develop and validate a novel approach to modeling fMRI activations in rich experiments with multiple stimuli or tasks. Rather than rely on a spatial correspondence across subjects to identify robust activations, the proposed methods will employ a notion of functional consistency, removing the need to assume spatial alignment among functional areas in different subjects. The resulting models of fMRI activation will also naturally enable studies of anatomical variability in homologous functional regions across subjects. The motivation for this work comes from visual fMRI studies that present subjects with several categories of visual stimuli. As fMRI studies move towards more complex experiments that include more stimuli, the space of possible brain responses grows exponentially, presenting a serious challenge for analysis methods. Explicit representations of fMRI activation patterns that enable exploratory search in the space of possible brain responses are at the core of this project. Computational models of brain activity based on such representations will significantly enrich the utility of fMRI for investigating the functional organization of the brain.

The research team will develop computational methods for fMRI analysis naturally suited for experiments with a multitude of stimuli. The approach is to model the space of all possible activation profiles, to search for stable clusters of activation profiles, and to characterize functionally homogeneous sets of brain locations associated with these clusters. A natural extension of the model will not only identify stable activation profiles but also group stimuli based on the similarity of the evoked activation profiles in the brain. Furthermore, this approach will yield a model of spatial variability of the detected functional areas, leading to better functionally-guided registration algorithms. The methods will be validated in a set of empirical experiments with a large number of visual stimuli in object perception and recognition tasks. The fMRI studies in this project will produce new insights into the functional organization of the ventral pathway of the visual system.

To model an individual's choices under uncertainty, theorists typically assume the choices made maximize the individual's utility. While frequently a good description of observed behavior, there are instances where people instead choose alternatives in proportion to their associated probabilities of reward. This probability matching behavior is sub-optimal. Probability matching behavior and optimal behavior would both result depending on the time available to make decisions (where more time produces more optimal decisions) if individuals base their choices on a sampling algorithm. In this Doctoral Dissertation Improvement grant, the PI will test whether such an algorithm is responsible for observed choices and, furthermore, whether people are optimally suboptimal (i.e., optimal in their decision regarding when to be more, or less, optimal.

To test these hypotheses, experimental subjects will be assessed in terms of how flexible they are at making tradeoffs between speed and accuracy in motor decisions under uncertainty and how generic decision processes are across decision domains. Subjects are then tested for whether their decisions under cognitive stress deteriorate to probability-matching, as predicted by the proposed algorithm. Finally, subjects will be tested using fMRI to determine whether one brain structure represents expected utility arising from different sources of uncertainty. This research holds promise for reconciling models of humans as ideal agents with established failures and limitations of human decision-making.

DESCRIPTION (provided by applicant): We humans are extraordinarily visual organisms, indeed a third of our cerebral cortex is devoted to processing visual information. The main starting point for much of this processing is a large region at the back of the brain called primary visual cortex, which holds a maplike representation of the visual world. This region connects to at least thirty other regions of the brain, each of which processes a different kind of visual information. Although most research focuses on the "forward" processing of visual information from primary visual cortex up to these higher stages of the visual brain, in fact it has been known for decades that there are at least as many connections that go in the other direction, from high visual areas "backward" down to primary visual cortex. Yet, the role of these "backward" connections in visual perception is not well understood. This proposal asks what those "backward" visual connections do, by investigating a surprising new phenomenon reported very recently by the PI's lab. In this new phenomenon, brain imaging (specifically, functional magnetic resonance imaging, or fMRI) was used to show that high-level information about the shape of an object represented presented in the peripheral visual field is present the center of the maplike representation in primary visual cortex. This finding is surprising because this information is in the "wrong" part of the visual map, so this information must be coming from higher level parts of the visual system, via the "backward" connections. This proposal uses fMRI, as well as behavioral methods and a method called transcranial magnetic stimulation (TMS) that allows us to disrupt this maplike representation, to ask why this "feedback" representation is found in visual cortex, whether it plays a causal role in visual perception, and exactly where it is within the maplike part of visual cortex. PUBLIC HEALTH RELEVANCE: This research is relevant to public health because it will tell us how the visual cortex works in humans. Disorders of vision are catastrophic, and we cannot help people with visual disorders without understanding how their brains process visual information. In particular, this work will be important for the future development of a neural prosthesis for people suffering from visual impairments, as well as for developing new training strategies for helping people recover from certain types of central visual disorders.

The last twenty years of brain imaging research has for the first time revealed the functional organization of the human brain in detail. We now know the function and location of dozens of regions of the brain that were not known 20 years ago. Many of these regions are involved in very specific components of cognition, such as the visual perception of color and motion, the visual recognition of faces, places, and bodies, and even high-level mental processes like understanding the meaning of sentence or thinking about what another person is thinking. Yet this tantalizing new map of the human mind and brain raises a pressing, unanswered question: How does all this precise functional organization get wired up in infancy and childhood? When and how does each little patch of cortex take on its distinctive adult function? Is the function of a given cortical region determined by the pre-existing connectivity of that region to the rest of the brain? How plastic is cortical organization in the event of early brain injury? To answer these questions, we will conduct extensive, longitudinal anatomical and functional scanning of children from birth through age 5. This work was not possible until now because it requires several very recent technical advances, including the new ?connectome? scanner at MGH that offers the resolution connectivity maps of the human brain of any scanner in the world, new methods of anatomical imaging in neonates, functional imaging in young infants, and longitudinal registration of brain images from the same person from birth to adulthood. These methods will enable us to test whether the cortical location of each functional region, when scanned at age 4-8, can be predicted from the distinctive connectivity of the same region (registered within subjects across age) at birth. By further including children with focal perinatal strokes, we can test the plasticity of specific cortical regions, versus white matter connections of those regions, in the eventual development of adult functional organization. This work will answer fundamental questions about how the human brain gets wired up over infancy and childhood that are also of great clinical relevance given the high prevalence of neurodevelopmental disorders in which this development goes awry.

The last ten years have witnessed an astonishing revolution in AI, with deep neural networks suddenly approaching human-level performance on problems like recognizing objects in an image and words in an audio recording. But impressive as these feats are, they fall far short of human-like intelligence. The critical gap between current AI and human intelligence is that, beyond just classifying patterns of input, humans build mental models of the world. This project begins with the problem of physical scene understanding: how one extracts not just the identities and locations of objects in the visual world, but also the physical properties of those objects, their positions and velocities, their relationships to each other, the forces acting upon them, and the effects of forces that could be exerted on them. It is hypothesized that humans represent this information in a structured mental model of the physical world, and use that model to predict what will happen next, much as the physics engine in a video game generates physically plausible future states of virtual worlds. To test this idea, computational models of physical scene understanding will be built and tested for their ability to predict future states of the physical world in a variety of scenarios. Performance of these models will then be compared to humans and to more traditional deep network models, both in terms of their accuracy on each task, and their patterns of errors. Computational models that incorporate structured representations of the physical world will then be tested against standard convolutional neural networks in their ability to explain neural responses of the human brain (using fMRI) and the monkey brain (using direct neural recording). These computational models will provide the first explicit theories of how physical scene understanding might work in the human brain, at the same time advancing the ability of AI systems to solve the same problems. Because the ability to understand and predict the physical world is essential for planning any action, this work is expected to help advance many technologies that require such planning, from robotics to self-driving cars to brain-machine interfaces. Each of the participating labs will also expand their established track records of recruiting, training, and mentoring women and under-represented minorities at the undergraduate, graduate, and postdoctoral levels. Finally, the collaborating laboratories will continue and increase their involvement in the dissemination of science to the general public, via public talks, web sites, and outreach activities.

Deep neural networks have revolutionized object recognition in computers as well as understanding of object recognition in the primate brain, but object recognition is just one aspect of vision, and the ventral stream is just one of many brain systems. Studying physical scene understanding is a step toward scaling this reverse-engineering approach up to the rest of the mind and brain. Predicting what will happen next and planning effective action requires understanding the physical basis and physical relationships in the visual world. Yet it is unknown how humans do this or how machines could. Both challenges are addressed in this project by the building of image computable, neurally mappable computational models of physical scene understanding and prediction (Thread I), and using these models as explicit hypotheses for how the brain might accomplish these tasks, which will then be tested with behavioral and neural data from humans (Thread II) and non-human primates (Thread III). This project aims to make a transformative leap in understanding: from small-scale, special-case models and isolated experimental tests to an integrated large-scale, general-purpose model of a major swathe of the primate brain, that functionally explains much of the immediate content of our perceptual experience in every scene that confronts us. The work will advance theory by developing the first image-computable models capable of human-level physical scene understanding and prediction. Beyond understanding of the mind and brain, this research is directly relevant to AI and robotics (which require physical scene understanding), and brain-machine interfaces (which require understanding of the relevant neural codes). For the broader research community, the project will a) develop public datasets, benchmark tasks, and challenges, b) host adversarial collaborations to address these challenges, and c) host interdisciplinary workshops linking research communities from psychology to AI to neuroscience to address the fundamental questions that span these fields.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.