Odelia Schwartz - US grants

DESCRIPTION (provided by applicant): Intellectual merit: A central question in neuroscience is understanding how cortical networks process complex natural stimuli. Neurophysiological studies and computational models have traditionally focused on simple stimuli, such as gratings and bars. While providing important insights, it is difficult to extrapolate from these studies to an understanding of the processing of more natural input. On the other hand, a main hurdle to making progress in the field is that natural scenes are complex and it is not clear what it is about a given scene that evokes a given neural response. To overcome this limitation and to push forward our understanding of cortical processing of natural inputs, we will make use of recent advances in understanding natural scene statistics to closely integrate theory and neurophysiological experiments. We posit that a key factor distinguishing natural images and movies from random scenes are joint statistical dependencies in space and time. Further, we hypothesize that visual neurons are sensitive to these dependencies. We will build a unified modeling framework of spatiotemporal contextual effects in neurons, which is determined by the statistical dependencies in scenes. Importantly, the predictions of the model will be used to guide neurophysiology experiments and to interpret the results. Using natural stimuli, we will measure effects of spatial, temporal, and spatiotemporal context in single neurons and in populations of cells, including determining how interactions between neurons contribute to contextual effects. We will record in primary visual cortex (V1) because it provides a solid background on which to base our experiments. We will conduct parallel recordings in extrastriate area V2 because previous work suggests that it may have different sensitivity to contextual information. The experimental results will validate and guide the modeling framework. Our approach will be a significant advance over previous scene statistics modeling work that has focused on explaining limited contextual physiology data for simple stimuli such as gratings, and will for the first time make full use of the power of scene statistics to answer a fundamental question. Most importantly, our work will make significant strides in elucidating how cortical circuits process natural scenes, within a theoretical framework that provides both predictive and explanatory power. Collaboration: The project will involve a collaborative effort between two young investigators with expertise in computational visual neuroscience and systems physiology;it combines state-ofthe- art algorithms from computational vision and technology for recording populations of neurons in early visual cortex. We will achieve our goal by closely integrating theory and model development with electrophysiological experiments, an approach fostered by the proximity of the two investigators. Broader Impacts: This proposal is expected to have broad impacts in five main areas. First, the work will have broad impact for basic, biomedical, and applied disciplines, including: studying other sensory systems under natural input;building superior visual aids;designing artificial systems;and advancing image and signal processing. Second, the data and stimuli will be made broadly available to the community through the CRCNS data sharing website. Third, the project will be used to train and mentor postdoctoral fellows to become independent research scientists. Fourth, the project will for the first time introduce students at Albert Einstein to the combination of theoretical and experimental approaches for solving fundamental questions in neuroscience. Finally, the project will be used as part of an outreach effort to expose local underrepresented high school students in the Bronx to exciting scientific research.

? DESCRIPTION (provided by applicant): In line with the strategic plan of the NEI, this project is focused on filling a profound gap in our understanding of neural mechanisms of visual perception. Specifically, we aim to understand how the adaptation of visual cortical circuits contributes to perception. Adaptation is a ubiquitous process by which neural processing and perception are dramatically influenced by recent visual inputs. However, the functional purpose of adaptation is poorly understood. Based on preliminary data, this project tests the hypothesis that visual adaptation instantiates a form of predictive coding, which is used to make unexpected events salient. We posit that cortical circuits learn the statistical structure of visua input in a manner that extends beyond previous fatigue- based descriptions of adaptation effects. This learning is used to discount expected features and signal novel ones. Our project will test this hypothesis through the collaborative effort of three investigators with expertise in human EEG, animal neurophysiology, and computational modeling. Aim 1 will assess the ability of cortical circuits to adapt to temporal sequences of input and to signal deviations from expected sequences. Aim 2 will evaluate the effect of stimulus uncertainty on adaptation and responses to novel events. Aim 3 will determine how adaptation dynamics and responses to novel stimuli are influenced by the temporal constancy of stimulus statistics. Each of these aims involves an experimental manipulation that yields distinct behavior from fatigue- based and predictive coding mechanisms. Thus, together our aims will provide a robust test of our core hypothesis, and provide a much richer understanding of the adaptive properties of cortical circuits. Results from our project will contribute to answering one of the continuing puzzles in visual research, which is to understand the functional purpose of adaptive mechanisms in visual perception.

Recent advances in artificial intelligence models of deep neural networks have led to tremendous progress in artificial systems that recognize objects in scenes, and in a host of other applications such as speech recognition, and robotics. Although deep neural networks often incorporate computations inspired by the brain, these have typically been applied in a fairly simple and restrictive manner, rather than based on more principled models of neural processing in the brain. Using vision as a paradigmatic example, this project proposes that artificial systems can benefit from integrating approaches that have been developed in biological models of neural processing of scenes. The biological models make use of contextual flexibility, whereby neurons are influenced in a rich way by the image structure that spatially surrounds a given object or feature. This flexibility is expected to improve task performance in deep neural networks, and to impact development of artificial systems that are more compatible with human cognition. The resulting framework, with its deep architecture spanning multiple layers of processing, will, in turn, make predictions about neural processing in the brain, which will impact the neuroscience and cognitive science communities.

This project focuses specifically on normalization, a nonlinear computation that is ubiquitous in the brain, and that has been shown to benefit task performance in deep neural networks. The project will develop more principled strategies for determining normalization in deep convolutional neural networks. The main focus will be on learning a form of flexible normalization based on scene statistics models of visual cortex. In this framework, normalization is recruited only to the degree that a visual input is inferred to contain statistical dependencies across space. Performance will be tested for classification and segmentation on large-scale image databases, and will also target tasks more suited to mid-level vision such as figure/ground judgment. This will result in better understanding of normalization nonlinearities in deep convolutional networks, and the implications of flexible normalization for task performance and generalization compared to other forms of normalization. Biologically, normalization is poorly understood beyond primary visual cortex. The models developed will help shed light on the equivalence of this inference for middle cortical areas, and make predictions about what image structure leads to recruitment of normalization. This project will also include launching of an interdisciplinary Deep Learning Discussion Group.