David L. Sheinberg - US grants

Representation and Computation in Natural Vision

In natural environments, objects are viewed under a wide variety of lighting conditions, poses, backgrounds, and juxtapositions with other objects. Artificial vision systems that are sufficiently invariant to accommodate such variations are never sufficiently selective. The rich structure of real images offers a multitude of chance arrangements, many of which cause systems to falsely detect an object that is not there. On the other hand, systems that are highly selective are at the same time highly prone to missed detections in the face of natural variability. The visual systems of humans and animals, in contrast, are able to see accurately under a wide range of viewing conditions--how is it that biological systems are both selective and invariant?

The pursuit of this question leads to an analogous question about complex cells and other invariant cell types that are ubiquitous in the ventral visual pathway. Their strength would appear to be their weakness: How is it possible for the visual system to build selectivity out of invariance? Models of complex cells suggest an explanation. Complex and other invariant cell types, by virtue of their nonlinear response characteristics, necessarily possess a functional connectivity whereby these cells become functionally connected to a generally small subset of their inputs. This commitment is circumstantial, inasmuch as it depends on the particular pattern in the receptive field. Functional connectivity is a demonstrable mathematical property of virtually all of the non-linear models put forward to date for complex-cell receptive-field properties. What is more, these observations lead to the conclusion that pairs of such cells that possess overlapping receptive fields will demonstrate a functional common input. This too is circumstantial, and in fact functional common input is high exactly when the patterns in the respective receptive fields "fit together"---correspond to pieces of a larger whole.

These observations suggest a solution to the dilemma of invariance versus selectivity: pieces that fit properly together generate a high degree of functional common input, which manifests itself by a statistical dependence between otherwise invariant representations, most likely in the form of partial synchrony, thereby signaling a composition of parts to cells deeper in the visual pathway.

In search of experimental confirmation of this proposed answer to the selectivity/invariance dilemma, the investigators employ new statistical and methodological techniques to study new questions about the receptive-field properties of invariant cells, and to measure new variables in the joint statistics of invariant cells with overlapping receptive fields.

DESCRIPTION (provided by applicant): Mounting evidence shows that functional connections between the cellular building blocks within the brain actively reorganize, even in the mature brain. Even so, this view is not yet universally accepted in part because tracking these changes in the awake, behaving animal is technically challenging. Further complicating the question of changes in single neuron properties with experience is the fact that cortex comprises many cell types, and recent data from the applicant's laboratory has revealed that plasticity may be expressed differentially across these populations. This proposal focuses on vision as a powerful model system for exploring the role that plasticity plays in normal brain function. The general hypothesis to be tested is that effective visual processing relies on experience driven, adaptive firing patterns of neurons within the inferior temporal cortex (IT), and that this experience leads to differential physiological changes in excitatory and inhibitory neurons. These changes, in turn, support measurable behavioral advantages. Although changes in neural responses are typically slow, artificial control of neural activity can induce modification more rapidly, and this modified activity can guide visually directed behavior. The proposed experiments will support efforts aimed at reviving or augmenting adaptive responses in higher-level visual areas. The proposal has three fundamental aims. The first aim is to clearly demonstrate impact of long-term familiarity on visual processing for multiple object classes. The strategy for accomplishing this aim will be to track performance in a speeded recognition tasks with well-known and trial unique stimuli. The second aim is to determine how visual experience affects stimulus encoding by neurons in anterior IT cortex. This will be achieved by tracking single neuron and small population activity by combining recording of activity across spatial scales and using carefully generated visual stimuli during the tasks developed in the first aim. The final aim is to directly manipulate neuron activity in temporal cortex to control plasticity. This aim will leverage optogenetic stimulation methods already in use in the applicant's laboratory to affect neural responses on single trials in order to induce the kinds of plasticity observed in the second aim. Together, the results of this work will help bridge the large literature on synaptic plasticity at he cellular level with visual behavior in primates. An important specific focus of these studies will e to identify stimulus, task, and physiological conditions under which both excitatory and inhibitory neurons adapt their responses through long-term experience, and to show how this plasticity can positively influence behavior. That visual experience can profoundly alter visual object representations in IT is of critical importance to efforts directed at repair of the visual system nd in understanding development disorders. Using an innovative set of tools and approaches, the projects in this proposal will emphasize the need to carefully track cellular activity in behaving animals, using complex and demanding real world tasks, with a level of resolution that will likely prove essential for future studies, and models, of higher brain function.

Last Modified Date: 08/01/08 Last Modified By: Daniel F. DeMenthon

Abstract
Normal vision is not static: time is a key dimension of the natural world we see. The eventual understanding of biological vision requires understanding the neural mechanisms used to recognize objects and actions over time. Thus the focus of the proposed research is to study how the primate visual system recognizes objects and actions in time sequences of images. A meta-goal of this project is to exploit the synergies between computational approaches and physiological experiments to lead to a better understanding of brain function and at the same time to develop better computer vision algorithms. Object recognition in time sequences of images presents a significant challenge for recognition systems, because it requires both selectivity to shape and invariance to changes of appearance in time..
This project will extend an existing computational model of the ventral stream by adding temporal dynamics in its model neurons and the ability to process video sequences. It will also expand a working model of the dorsal stream to understand the relative roles that it and the ventral stream play in dynamic visual recognition. At the same time, recordings from single units, and multiple single units, from high level visual areas including IT and regions of the STS will be made in order to characterize the tuning of single neurons to the shape dynamics of specific image sequences. By combining modeling and physiology, this work will search for a computational explanation for how the higher areas of the visual cortex recognize objects and actions over time and how they can learn.
This integrative effort, which is focused on processing of dynamic perceptual information, can have a significant and direct impact on current theories of autism, dyslexia, and effects of stroke, in addition to directly guiding modeling and engineering efforts in computer vision. The proposed research is tightly coupled to education and teaching, and resources used in the research, including databases of videos, visual stimuli, the modeling software and the experimental data will be made available to the broad scientific community. Information on the project and its progress will be available at http://cbcl.mit.edu/projects/NSF-CRCNS/index.html

? DESCRIPTION (provided by applicant): Our Interdisciplinary Predoctoral Neuroscience Training Program strives to provide individualized, high quality training to predoctoral students interested in pursuing scientific research careers in the biological and biomedical sciences. This training grant will support students in their first two years of graduate studies, before they star their dissertation research. We request support for 8 students per year. Graduate students in our program receive broad, multi-disciplinary training that spans many levels of inquiry, from genes through cognition, and emphasizes concepts, methodologies, quantitative skills, and sophisticated analysis of the primary literature. Our core curriculum consists of team-taught graduate courses, seminars, and workshops that provide a strong scientific foundation in neuroscience and develop skills that are essential for successful, independent research careers in neuroscience, such as effective science writing and oral presentation, knowledge of scientific review processes, and training in ethics. We have introduced new initiatives to expose students to translational and clinical neuroscience with our Bench to Bedside seminar series. On average, students in our program finish their PhD in 5.35 years, and the majority of our alumni continue their careers in science-related fields including academic or industry science positions. We foster an environment unconstrained by traditional discipline boundaries and where graduate students are encouraged to work at the interfaces of these disciplines. The training program includes 34 core participating faculty and ~50 predoctoral trainees. The faculty trainers are drawn from seven different Brown University departments: Neuroscience; Cognitive, Linguistic, and Psychological Sciences; Molecular Biology, Cell Biology, and Biochemistry; Engineering; Molecular Pharmacology, Physiology and Biotechnology; Biostatistics; and Neurosurgery. They are a distinguished and energetic group of brain scientists that collectively cover the spectrum of modern neuroscience research: they work with a wide variety of model organisms, from worms to humans, and use an array of modern neuroscience techniques, including functional MRI, applications of robotics and neuroprosthetics, optogenetics, advanced in vivo and in vitro electrophysiological recordings, mouse transgenics, behavioral studies, molecular manipulations of neuronal genes, functional proteomics, and human genome-wide association studies. We encourage and facilitate collaborations between labs as well as research in computational and translational neuroscience that typically reside at the interface of disciplines. Key features of the Neuroscience Graduate Program at Brown include: Excellence in research along with excellence in education and mentorship; a history of interdisciplinary and translational research; rigorous training in experimental design and quantitative methods, and an environment of highly productive labs where graduate students are equal partners in the research process.

Abstract/Summary Tracking the movements of objects and parts of objects - referred to as pose estimation - is critical for understanding the mechanisms underlying complex behavior. Characterizing dynamic behaviors of animals (and other systems) is central to many disciplines, including computer science, physics, ethology, kinesiology, and sports medicine. Here we focus on neuroscience, where linking brain activity to associated dynamic behaviors is critical for both understanding normal function as well as effects of injury, disease, or degeneration. Invasive methods for measuring behavior are highly accurate, but require placement of sensors that may themselves interact with behavior and which may be susceptible to deterioration or infection. Video provides a non-invasive approach to characterizing behavior over time. Extracting behavior from video streams has, historically, been a slow and laborious process. Recent work in machine learning and artificial neural networks (ANNs), though, has revolutionized this process, making the analysis of complex video far easier and more accurate. While these systems are highly flexible, they were not designed for real time use, meaning that large video files must first be stored to disk for subsequent analysis. This poses two problems that this proposal will attempt to address. First, there is significant cost and management challenges associated with storage of large video stores, forming a practical barrier for adoption of this important technology for characterizing behavior. Second, estimates related to behavioral state are not available in real time so they cannot be used to control the experiment. We will develop a research methodology for ?closing the loop?, by taking the networks trained by an existing and highly successful markerless object tracking system (DeepLabCut) and optimizing them for real time inference. After the system is functional, verified, and benchmarked, it will be shared with the community through open source repositories.

PROJECT SUMMARY / ABSTRACT Our Interdisciplinary Predoctoral Neuroscience Training Program strives to provide individualized, high quality training to predoctoral students interested in pursuing scientific careers in the biological and biomedical sciences. This training grant will support 8 students in their first two years of graduate studies, before they start their dissertation research. Graduate students in our program receive broad, multi-disciplinary training that spans many levels of inquiry, from genes through cognition, and emphasizes concepts, methodologies, experimental design, quantitative skills and program, and sophisticated analysis of the primary literature. Our core curriculum consists of graduate only courses, seminars, and workshops that provide a strong scientific foundation in neuroscience and develop skills that are essential for successful, independent research careers in neuroscience, such as effective science writing and oral presentation, knowledge of scientific review processes, and training in ethics. New initiatives include a revised advising system, more structured program evaluation, and greatly expanded quantitative training. We foster an environment unconstrained by traditional discipline boundaries, where graduate students are encouraged to work at the interfaces of these disciplines. The training program includes 39 core participating faculty and ~60 predoctoral trainees. The faculty trainers are drawn from eight different Brown University departments: Neuroscience; Cognitive, Linguistic, and Psychological Sciences; Molecular Biology, Cell Biology, and Biochemistry; Engineering; Molecular Pharmacology, Physiology and Biotechnology; Biostatistics; Neurology; and Neurosurgery. They are a distinguished and energetic group of brain scientists that collectively cover the spectrum of modern neuroscience research: they work with a wide variety of model organisms, from worms to humans, and use an array of modern neuroscience techniques, including functional MRI, applications of robotics and neuroprosthetics, optogenetics, advanced in vivo and in vitro electrophysiological recordings, mouse transgenics, behavioral studies, molecular manipulations of neuronal genes, functional proteomics, and human genome-wide association studies. We encourage and facilitate collaborations between labs as well as research in computational and translational neuroscience that typically reside at the interface of disciplines. Key features of the Neuroscience Graduate Program at Brown include: Excellence in research along with excellence in education and mentorship; a focused effort on addressing shortcomings related to diversity and equity across our program, including recruitment and retention of students as well as broad representation of trainer backgrounds; a history of interdisciplinary and translational research; rigorous training in experimental design and quantitative methods, and an environment of highly productive labs where graduate students are equal partners in the research process.

Abstract/Summary The goal of this project is to develop an animal model for visual simulation. In particular, we aim to show that animals, like humans, can engage in extended visualization of events not present in the external world. This capacity is essential, as it allow animals to test strategies and reason about the world using internal visual machinery. Until now, almost all exploration of visual simulation has been carried out in humans, as verbal instructions are the most common way to evoke controlled imagined experiences. Here we introduce a novel task, initially tested in humans, to non-human primate subjects. In Aim 1, we will show that their performance in this task mirrors that of human subjects. We will then have these trained animal subjects participate in awake fMRI studies while performing this task, and associated controls, in order to test the hypothesis that the same visual circuits involved in active perception of dynamic visual events are recruited during internally driven visual simulation of these events (Aim 2). Successful completion of these aims will provide a key starting point for the first systematic neurophysiological investigations of visual mental simulation at the single neuron level. This has significant implications for understanding the rich interplay between the visual system and higher mental function in both normal perception and neurological disease.