Thomas Naselaris - US grants

DESCRIPTION (provided by applicant): Mental imagery is a salient part of mental awareness but very little is understood about how visual percepts are generated without retinal input, or how visual features that are known to be an important part of visual representation drive neural activity during mental imagery. Our long-term goal is to provide clinicians with the ability to objectively interpret mental images by accessing underlying neural activity. The objective of the current work is to develop a basic understanding of the similarities and differences between the representation of visual features in veridical and mental images. Our central hypothesis is that the mechanisms for representing visual features during perception are fundamentally conserved during mental imagery and that receptive fields that link activity to veridical images should predict activity evoked by mental imagery. Nonetheless, mental images are clearly distinguishable from veridical images and we consider three potential sources of difference: (1) The potential for exaggerated effects of attention on mental imagery; (2) The predominate influence of feedback connections from high-level visual areas with large receptive fields (relative to the retina) during mental imagery; (3) Differences between the neural processes of generating mental images and the physical processes that generate retinal images. Two Specific Aims are proposed that will be pursued using an innovative new approach for analyzing functional MRI signals that is based upon voxel-wise modeling of receptive fields. Under this approach, a separate predictive model is constructed for each and every voxel in the acquired volumes. The model links activity measured in a voxel directly to specific visual features, including spatial frequency, orientation, object category, and object location. The models can then be used to decode perceived or recalled scenes from measured brain activity. We expect that our contribution will be an advance in our understanding of the specific factors that determine the degree of consistency between activity during imagery and perception, as well as a significant advance in our ability to quantitatively model the high-level visual areas where activity is most consistent. This contribution will be significant because it will take us several necessary steps toward the development of imagery receptive fields-predictive receptive field models that explain how the visual features in a scene drive activity when the scene is recalled in the form of a mental image. A receptive field model for mental imagery would place within reach a decoding algorithm for objectively interpreting and even pictorially reconstructing mental images.

PROJECT SUMMARY Our knowledge of signal processing in various parts of the human brain has been heavily influenced by non- invasive functional magnetic resonance imaging (fMRI) experiments. FMRI infers the location and selectivity of neural activity from vascular signals. However, brain circuits are much more complex than regional differences in neuronal selectivity. Specifically, the largest part of the brain (neocortex) accounts for up to 80% of the brain volume and is divided into six distinct layers. Specific computations, e.g., local processing vs. feedforward inputs vs. vs. feedback inputs, are done in specific cortical laminae. Thus, if high-resolution layer-specific fMRI is shown to reflect the repertoire of neural computations performed across these cortical layers, it would be an invaluable refinement to non-invasive imaging. However, despite the widespread usage of low-resolution fMRI, a detailed understanding of how neural activity generates vascular responses remains unknown. The goal of this project is to elucidate the link between neural and vascular signals across laminae by combining two-photon imaging of neural and vascular responses with ultra-high-field (UHF) fMRI. Experiments will use sensory visual stimuli that induce layer-specific responses. In cat primary visual cortex (V1), which has a functional architecture (e.g., maps for stimulus orientation) similar to human V1, we will measure neural activity (synaptic and spiking) with single-cell resolution together with vascular signals (blood flow, blood volume, and oxygenation) in individual vessels across the entire cortical thickness. We will also perform UHF lamina-specific fMRI in cat (9.4 and16.4 T) and human (7 and 10.5 T) V1 to relate fMRI signals to the single- vessel responses. Lastly, we will develop a model to relate lamina-specific vascular signals to neural activity. In Aim 1, we test the hypothesis that vascular signals selective for stimulus orientation are present in cortical layers 2/3 (and 5/6) while untuned responses occur in layer 4 and pial vessels. Grating visual stimuli will be used, while varying orientation and eye preference (ocular dominance) systematically. Since binocular integration is stronger outside layer 4, eye preference vascular signals should be most prominent in layer 4. In Aim 2, we will test the hypothesis that in any given cortical lamina, glutamate release in regions around an individual blood vessel best accounts for the selectivity of vascular responses compared to spiking activity?in terms of the preferred stimulus orientation and tuning width. Aim 3 is to build a computational model to determine effective minimum voxel size for BOLD fMRI. The model will be tested against simultaneously measured vascular and neural activity to natural scene stimuli using two-photon imaging. If the source signals at the finest spatial scales have laminar specificity, we can correlate laminar-specific fMRI signals to differences in neural processing. To our knowledge, this is the first study that brings together such a wide repertoire of approaches into a single project to understand the neural and laminar basis of fMRI.

Cognitive Computational Neuroscience (CCN) is an annual scientific meeting for neuroscientists characterizing the neural computations that underlie complex behavior. The goal is to develop computationally defined models of brain information processing that explain rich measurements of brain activity and behavior. Such models will ultimately have to perform feats of intelligence such as perception, internal modelling and memory of the environment, decision-making, planning, action, and motor control under naturalistic conditions. Historically, different disciplines have met subsets of these goals. Cognitive science has developed computational models at the cognitive level to explain aspects of complex behavior. Computational neuroscience has developed neurobiologically plausible computational models to explain neuronal responses to sensory stimuli and certain low-dimensional decision, memory, and control processes. Cognitive neuroscience has mapped a broad range of cognitive processes onto brain regions. Artificial intelligence has developed models that perform feats of intelligence. The community must now put the pieces of the puzzle together, and CCN is unique in its focus on the intersection between these fields. CCN is envisioned not only as an engine for advancing research, but as a vehicle for making broader impacts on education and society. As evidenced by the recent trend of major corporate acquisitions of AI startups founded by neuroscientists, biological inspiration for electronics and software development is a growing trend with significant economic implications. In its early stages, the broader impact focus of CCN will be on increasing the visibility of women and scientists from underrepresented populations via speaking opportunities and travel awards. In addition, representation on women on the female fractions on the steering and advisory committees exceed those typical in relevant fields, without compromise in qualifications. Conferences will include hands-on tutorials, and materials from these will propagate to various university curricula.

A central goal of neuroscience is to understand how vast populations of neurons give rise to complex behavior. Today, advances in various domains offer tangible possibilities to make fundamental conceptual breakthroughs. From an experimental point of view, neural recording technologies, such as high-resolution fMRI, dense recording arrays, magnetoencephalography (MEG), and calcium imaging, now provide opportunities to observe neural activity at unprecedented resolution and scale. At the same time, research in cognitive science has become increasingly sophisticated in identifying computational principles that may serve as the basis for human cognition, and machine learning and artificial intelligence have made great strides in building models to autonomously solve complex cognitive tasks. However, interactions among these distinct disciplines remain rare. This new conference may stimulate unifying frameworks that fully realize the cross-disciplinary potential of these individual advances. In more concrete terms, the goal of CCN is to create and foster a community that will develop models of brain information processing with several key features. These models should (1) be fully computationally defined and implemented in computer simulations; (2) be neurobiologically plausible; (3) explain measurements of brain activity (and continue to do so as spatiotemporal resolution and scale improve); (4) explain behavior for naturalistic stimuli and tasks; and (5) perform feats of intelligence such as recognition, internal modelling and representation of the environment, decision-making, planning, action, and motor control. Such models currently do not exist and are unlikely to emerge without greatly improved cross-disciplinary engagement.

This project will provide three-years of support for the Conference on Cognitive Computational Neuroscience (CCN). This conference provides an annual scientific meeting for neuroscientists whose goal is to develop computationally defined models of brain information processing that explain rich measurements of brain activity and behavior. Historically, different disciplines have met subsets of these goals: Cognitive science has developed computational models at the cognitive level; computational neuroscience has developed neurobiologically plausible computational models at lower levels; cognitive neuroscience has mapped processes onto brain regions; and artificial intelligence has developed synthetic systems. CCN is unique in its focus on the intersection between these fields. In addition to advancing research, CCN seeks to contribute to the growing commercial use of biologically inspired hardware and software in Artificial Intelligence as well as being a vehicle for broadly impacting education and society. One particular focus of CCN is increasing the visibility of women and scientists from underrepresented populations via speaking opportunities. This award will partially support travel grants for this purpose. The conference will also include hands-on tutorials, and materials from these will propagate to various university curricula. The award will support video recordings of the tutorials and talks. These recordings will be made publicly available on the website to increase the broader impact of the conference to the wider community and those unable to attend.

A central goal of neuroscience is to understand how vast populations of neurons give rise to complex behavior. Today, advances in various domains offer tangible possibilities to make fundamental conceptual breakthroughs. Modern neural recording technologies now provide opportunities to observe neural activity at unprecedented resolution and scale. At the same time, research in cognitive science has become increasingly sophisticated in identifying computational principles that may serve as the basis for human cognition, and machine learning and artificial intelligence have made great strides in building models to autonomously solve complex cognitive tasks. However, interactions among these distinct disciplines remain rare. This new conference may stimulate unifying frameworks that fully realize the cross-disciplinary potential of these individual advances. Concretely, the goal of CCN is to create and foster a community that will develop models of brain information processing with several key features. These models should (1) be fully computationally defined and implemented in computer simulations; (2) be neurobiologically plausible; (3) explain measurements of brain activity (and continue to do so as spatiotemporal resolution and scale improve); (4) explain behavior in the context of naturalistic stimuli and tasks; and (5) perform feats of intelligence such as recognition, internal modelling and representation of the environment, decision-making, planning, action, and motor control. Such models currently do not exist and are unlikely to emerge without greatly improved cross-disciplinary engagement.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Machine learning technologies have the potential to radically transform the study of the human brain, but require far more data than is typically collected during conventional neuroscience experiments. The goal of this project is to drive the application of ML techniques to neuroscience research by generating a massive dataset of brain responses from the human visual system. The resulting dataset will be freely available to scientists, educators, and students. Through a yearly modeling competition, neuroscientists will gain experience in the application of advanced computational methods and ML researchers will gain a deeper understanding of the challenges and complexities of the human brain. Results of the modeling competition will be presented at an annual conference attended by both machine learning and neuroscience researchers and students, providing an opportunity for the two groups to interact and discuss approaches. This project will foster open collaboration between neuroscientists and artificial intelligence researchers and a culture of sharing data, ideas, and progress.

The long-term goal of this work is to generate data that will lead to the development of experimentally validated and computationally powerful models of the human visual system. The project leaders will use high-field (7 Tesla) functional magnetic resonance imaging (fMRI) to measure brain responses to a broad sampling of natural images in human observers. The specific objectives are as follows: (1) Acquire, pre-process, and distribute a massive, high-resolution fMRI dataset that exploits state-of-the-art imaging techniques. The dataset will include multiple samples of brain responses to roughly eighty thousand photographs drawn from an image collection that is widely used by the ML community. (2) Establish and host an annual competition for modeling this rich dataset at the conference on Cognitive Computational Neuroscience. (3) Bridge the gap between ML architectures and the human brain by testing new ML-inspired architectures as models of the visual system. The project leaders will focus specifically on recent developments in ML that suggest new hypotheses about the dorsal visual stream.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.