James J. DiCarlo - US grants

DESCRIPTION (provided by applicant): Visual object recognition is central to our behavior, and knowledge of the underlying brain mechanisms is critical to understanding human visual perception and memory. The key problem is creation of selectivity for object identity that tolerates changes in an object's retinal image, such as changes in position and size. The primate brain appears to construct this selectivity in the ventral visual stream because neuronal responses in the highest area of that stream--the anterior inferotemporal cortex (AIT)--show shape selectivity that can tolerate position and size changes. Yet, we do not understand these key neuronal properties--reports of AIT tolerance are limited and inconsistent, and recent studies show that it can be very restricted. Thus, the goals of this proposal are an understanding of key factors likely to determine AIT position and size tolerance, and to determine if AIT tolerance can explain behavioral tolerance. Our first aim is to systematically determine the position and size tolerance of AIT neuronal shape selectivity for a range of object sets and object training histories. We will establish the relationship of selectivity and AIT position and size tolerance, the interaction of AIT position and size tolerance, and the effect of object-specific training on these relationships. These data will establish neuronal tolerance at the highest level of the primate visual system and provide a much-needed foundation for further study. The mechanisms that might underlie position and size tolerance fall into two broad classes: (1) automatic generalization; and (2) tolerance learned by experiencing objects across changes in position and size. Our second aim is to determine if position- or size-specific object experience have substantial effects on the position or size tolerance of AIT shape selectivity. Because this has not been examined, any result would be extremely informative in constraining mechanisms and guiding future studies. Although it is thought that AIT tolerance underlies behavioral tolerance, this has not been systematically examined. Our third aim is to determine if the position and size tolerance of object identification can be explained by the tolerance of AIT neuronal shape selectivity. This is a vital to understanding the link between high-level, ventral stream neuronal responses and visual object identification.

This project exploits advances in parallel computing hardware and a neuroscience-informed perspective to design next-generation computer vision algorithms that aim to match a human's ability to recognize objects. The human brain has superlative visual object recognition abilities -- humans can effortlessly identify and categorize tens of thousands of objects with high accuracy in a fraction of a second -- and a stronger connection between neuroscience and computer vision has driven new progress on machine algorithms. However, these models have not yet achieved robust, human-level object recognition in part because the number of possible "bio-inspired" model configurations is enormous. Powerful models hidden in this model class have yet to be systematically characterized and the correct biological model is not known.

To break through this barrier, this project will leverage newly available computational tools to undertake a systematic exploration of the bio-inspired model class by using a high-throughput approach in which millions of candidate models are generated and screened for desirable object recognition properties (Objective 1). To drive this systematic search, the project will create and employ a suite of benchmark vision tasks and performance "report cards" that operationally define what constitutes a good visual image representation for object recognition (Objective 2). The highest performing visual representations harvested from these ongoing high-throughput searches will be used: for applications in other machine vision domains, to generate new experimental predictions, and to determine the underlying computing motifs that enable this high performance (Objective 3). Preliminary results show that this approach already yields algorithms that exceed state-of-the-art performance in object recognition tasks and generalize to other visual tasks.

As the scale of available computational power continues to expand, this approach holds great potential to rapidly accelerate progress in computer vision, neuroscience, and cognitive science: it will create a large-scale "laboratory" for testing neuroscience ideas within the domain of computer vision; it will generate new, testable computational hypotheses to guide neuroscience experiments; it will produce a new kind of multidimensional image challenge suite that will be a rallying point for computer models, neuronal population studies, and behavioral investigations; and it could unleash a host of new applications.

DESCRIPTION (provided by applicant): Visual object recognition is central to quality of life in health and disease, but it is not understood at a deep, mechanistic level. For example, while primate inferior temporal cortex (IT) is likely a key neuronal processing bottleneck, we still have only a dim understanding of its causal role at a fine spatial and temporal grain. This exploratory proposal (R21) aims to deploy, characterize, and behaviorally validate novel tools to produce spatially precise, temporally delimited silencing of neuronal activity in the IT cortex of the awak, behaving primate. Concretely, we want to choose a mm-scale location on a magnetic resonance image of a non-human primate brain, and then ask: what is the importance of normally-evoked neuronal activity at that location in supporting a given behavioral task? Rather than try to inject neuronal signals, our strategy is to develop methods to briefly (10-300 ms) block the neuronal activity that normally intervenes between visual stimulus onset and the animal's reaction time. To that end, this exploratory proposal has two synergistic aims: First, our preliminary results show that virally delivered optically-gated silencing molecules can indeed produce strong silencing of neuronal activity in IT cortex, but we have little understanding of the reliability, spatial extent and temporal limits of this silencing. Thus, we will (Aim 1) make x-ray targeted viral injections, followed by x-ray targeted optical fiber implantation, and spatially precise (~10 um) maps of neuronal silencing (or enhancement) effects in and around the optical fiber tip at multiple sites in IT cortex. The expected outcome is a spatiotemporal map of light-induced neuronal silencing around the optical fiber tip, and its dependence on light intensity, duration and latency. Second, we do not know if optical silencing of IT sub-regions leads to measurable behavioral effects on object recognition tasks. Thus, we have developed recognition tasks that are likely to be affected by IT silencing, and we have already discovered that pharmacological neuronal silencing at specific IT sub-regions (muscimol) leads to behavioral deficits in at least one recognition task (but not all such tasks). We now aim (Aim2) to test the ability of optical silencing tools to produce behavioral deficits in that same task at those same locations. The expected outcome is a demonstration that optical silencing of IT sub-regions can produce specific behavioral recognition deficits, as well as a comparison with pharmacologically induced deficits. If successful, the proposed work will enable entirely new lines of interventional work to systematically test the causal role of IT cortex in visual object recognition, and will contribute o a still nascent, but promising toolbox of optical techniques for systems-level questions in primates.

DESCRIPTION (provided by applicant): The Brain and Cognitive Sciences Graduate Program at the Massachusetts Institute of Technology requests renewal of its major training grant. The department is organized to promote interdisciplinary training and research in neuroscience and behavior, approached with the experimental power of modern molecular and cellular neuroscience, systems neuroscience, and cognitive science, combined with the theoretical strength of computational neuroscience and artificial intelligence. Trainees begin laboratory work through lab rotations in the first two terms and subsequently join a laboratory, working on problems in learning and memory, neural development, vision, motor control or brain disorders and diseases. Required course work can be completed in two to three years, with a two-term sequence of core courses in the first year, a quantitative methods course, and a flexible array of graduate lecture courses and seminar classes. The qualifying exam consists of written and oral components of an interdisciplinary NIH/NSF style grant proposal. Annual research reports and annual committee meetings are required, and mark the student's progress in research through completion of a thesis. Multiple presentations at professional meetings and journal publications are typically expected of a dissertation. Most students continue in research careers, armed with skills that typically span multiple theoretical and experimental approaches comprising molecular/cellular neuroscience, systems neuroscience, cognitive neuroscience, psychophysics, behavior and computation. Trainees will, in general, have strong backgrounds in the natural sciences (e.g., undergraduate majors in biology, chemistry, physics, mathematics, or electrical engineering). Occasional trainees will already hold a master's degree in another field. Candidates for the graduate program will be chosen by the department Graduate Committee constituted for the purpose of overseeing this program and will be evaluated on the basis of interviews, talent for research as demonstrated by past performance, letters of recommendation, grades, and GRE scores. Funds are requested for five years to support 12 predoctoral trainees per year.

View-invariant object recognition is a complex cognitive task that is critical to everyday functioning. A key neural correlate of high-level object recognition is inferior temporal (IT) cortex, a brain area present in both humans and non-human primates. Recent advances in visual systems neuroscience have begun to uncover how images are encoded in the adult IT object representation, however the learning rules by which high level visual areas (especially IT) develop remain mysterious, with both the magnitude and qualitative nature of developmental changes remaining almost completely unknown ? in part because, over the last thirty years, there have been practically no studies of spiking neural responses in the higher ventral cortical areas of developing primates. There is thus a significant gap in our understanding of how visual development proceeds. This exploratory proposal aims to characterize how representation in higher primate visual cortex changes during development. We first aim (Aim 1) to implant chronic electrode arrays to record hundreds of IT neuronal sites in response to thousands of image stimuli in awake behaving juvenile macaques. These data will comprise a snapshot of the developing primate visual representation, and will be particularly powerful because we have already extensively measured adult monkey IT using the same stimuli and methods. By comparing juvenile and adult neuronal responses at both single site and population levels, we will obtain a unprecedentedly large-scale and detailed picture of the neural correlates of high-level visual development (Aim 2). Aims 1 and 2 are exploratory, but potentially transformative ? they will result in publicly available neuronal IT development benchmarks against which any proposed model of high level visual development can be rigorously tested, and will spur the development of those models in our lab and others. In that context, we will also seek (Aim 3) to improve known semi- and un-supervised learning rules from the computer vision and computational neuroscience literature, and to compare them to both recent high-performing (but biologically implausible) supervised models as well to the rich developmental measurements obtained in Aims 1 and 2. Establishing experimental and surgical procedures for juvenile array recordings will create the future opportunity to observe changes in high level neural visual representations while experience is manipulated in early development, and will enable experiments in other sensory, motor, or decision making domains. If successful, the proposed work will yield a deeper understanding of the principles underlying visual cortex development, understanding which will in turn be helpful for treating neurodevelopmental disorders that implicate cortical circuits, including amblyopia and autism.

ADMINISTRATIVE CORE: Project Summary The Administrative Core ensures the three service Cores (Machine Core, Electronics Core and Imaging Core) are efficiently run, coordinated, communicated, and evolved as needed to meet the overall goals of the NEI Core grant. Most importantly, the Administrative Core supervises, mentors, and administratively supports the personnel in the service Cores to keep the Core personnel working toward the specific aims of each of the service Cores and provides opportunities for additional training of those personnel as needed for those aims. The Administrative Core ensures the availability and equitable use of the service Cores and develops and deploys software to assist in this effort. The Administrative Core upholds a culture of transparency and collaboration by communicating the ever-evolving Core services to keep all investigators up to date with the technologies and capabilities of each Core, communicating the projects in the service Cores, and facilitating dissemination of designs and know how. Finally, the Administrative Core tracks changes in research needs of the Core NEI Investigators and adapts the Cores to meet those ever-evolving needs. Because these efforts keep the Cores focused on their overall mission, all of the proposed activities were previously performed on an ad hoc basis. However, the formal addition of the Administrative Core in this renewal application will enhance these activities and thus enhance the impact of the overall Core.

The last ten years have witnessed an astonishing revolution in AI, with deep neural networks suddenly approaching human-level performance on problems like recognizing objects in an image and words in an audio recording. But impressive as these feats are, they fall far short of human-like intelligence. The critical gap between current AI and human intelligence is that, beyond just classifying patterns of input, humans build mental models of the world. This project begins with the problem of physical scene understanding: how one extracts not just the identities and locations of objects in the visual world, but also the physical properties of those objects, their positions and velocities, their relationships to each other, the forces acting upon them, and the effects of forces that could be exerted on them. It is hypothesized that humans represent this information in a structured mental model of the physical world, and use that model to predict what will happen next, much as the physics engine in a video game generates physically plausible future states of virtual worlds. To test this idea, computational models of physical scene understanding will be built and tested for their ability to predict future states of the physical world in a variety of scenarios. Performance of these models will then be compared to humans and to more traditional deep network models, both in terms of their accuracy on each task, and their patterns of errors. Computational models that incorporate structured representations of the physical world will then be tested against standard convolutional neural networks in their ability to explain neural responses of the human brain (using fMRI) and the monkey brain (using direct neural recording). These computational models will provide the first explicit theories of how physical scene understanding might work in the human brain, at the same time advancing the ability of AI systems to solve the same problems. Because the ability to understand and predict the physical world is essential for planning any action, this work is expected to help advance many technologies that require such planning, from robotics to self-driving cars to brain-machine interfaces. Each of the participating labs will also expand their established track records of recruiting, training, and mentoring women and under-represented minorities at the undergraduate, graduate, and postdoctoral levels. Finally, the collaborating laboratories will continue and increase their involvement in the dissemination of science to the general public, via public talks, web sites, and outreach activities.

Deep neural networks have revolutionized object recognition in computers as well as understanding of object recognition in the primate brain, but object recognition is just one aspect of vision, and the ventral stream is just one of many brain systems. Studying physical scene understanding is a step toward scaling this reverse-engineering approach up to the rest of the mind and brain. Predicting what will happen next and planning effective action requires understanding the physical basis and physical relationships in the visual world. Yet it is unknown how humans do this or how machines could. Both challenges are addressed in this project by the building of image computable, neurally mappable computational models of physical scene understanding and prediction (Thread I), and using these models as explicit hypotheses for how the brain might accomplish these tasks, which will then be tested with behavioral and neural data from humans (Thread II) and non-human primates (Thread III). This project aims to make a transformative leap in understanding: from small-scale, special-case models and isolated experimental tests to an integrated large-scale, general-purpose model of a major swathe of the primate brain, that functionally explains much of the immediate content of our perceptual experience in every scene that confronts us. The work will advance theory by developing the first image-computable models capable of human-level physical scene understanding and prediction. Beyond understanding of the mind and brain, this research is directly relevant to AI and robotics (which require physical scene understanding), and brain-machine interfaces (which require understanding of the relevant neural codes). For the broader research community, the project will a) develop public datasets, benchmark tasks, and challenges, b) host adversarial collaborations to address these challenges, and c) host interdisciplinary workshops linking research communities from psychology to AI to neuroscience to address the fundamental questions that span these fields.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.