Jack L. Gallant - US grants

Our long term goal is to understand the neural basis of complex form vision as it operates under natural viewing conditions. Decades of vision research using simple stimuli and controlled viewing conditions have produced sophisticated models of processing in the early visual system. We propose to evaluate these models directly by recording responses under conditions simulating natural vision. In Aim 1 we propose to acquire data in area V1 using both naturalistic and more standard conventional stimuli. Quantitative methods will be used to estimate cells' spatio-temporal filtering properties from these data. The experiments and analyses will reveal how are V1 represents and processes information during natural vision. They will also reveal how these functions differ from those predicted from experiments using conventional simple stimuli. In Aim 2, we will evaluate a range of functional computational models of area V1 to identify those models that can account for responses observed during simulated natural vision. This will be accomplished by direct statistical comparison between models. Successful completion of this Aim will result in the first computational model(s) that can account for the spatio-temporal responses of V1 neurons during natural vision. In Aim 3 we will assess the modulatory influence of extra-retinal factors such as attention during simulated natural vision. We propose to do this by combining standard match-to-sample behavioral tasks with our naturalistic stimuli and our quantitative receptive field estimation methods. This will provide a sensitive measure of potential extra-retinal effects in area V1. Experiments on natural vision are important because they will allow us to evaluate our current theories of human visual cortical function and identify areas in which our understanding is weak. They will also allow us to address complex aspects of form vision that cannot easily be approached using more conventional methods.

DESCRIPTION (provided by applicant): This proposal addresses the way the visual system processes complex shape. We focus on two intermediate visual areas, V2 and V4, located in the ventral processing stream immediately beyond primary visual cortex (area VI). These areas serve as the major input stages for higher-order shape processing areas in the temporal cortex. We propose neurophysiological experiments to investigate the way that shape is represented in these areas and the way that attention modulates these representations. Shape is difficult to describe and parameterize, so previous neurophysiological studies of shape processing have utilized simple, regular shapes that are experimentally convenient. However, intermediate shape processing is highly nonlinear, so results obtained with reduced stimulus sets may not generalize to other stimuli. We therefore propose to use both complex, natural stimuli and simpler stimuli such as gratings. To facilitate this, we are developing novel nonlinear regression algorithms to estimate the stimulus-response mapping functions of neurons in V2 and V4. The underlying shape dimensions represented therein can then be determined by applying visualization algorithms (developed in our laboratory) to the stimulus-response mapping functions estimated for single neurons. In another series of experiments we plan to investigate how extrastriate visual areas integrate information from earlier sensory areas. Finally, we propose to examine how visual attention affects shape representations in V2 and V4. We will accomplish this by quantifying the effects of selective attention to a specific shape (feature attention) and attention directed toward a specific location in space (spatial attention) on neuronal tuning curves. Successful completion of these projects will provide critical information to aid in development of quantitative computational models of shape processing in intermediate vision.

Proposal No: 0749056
PI: Yang Dan

Award Abstract:

This award supports the preparation and sharing of computational neuroscience data as part of an exploratory activity aimed at catalyzing rapid and innovative advances in computational neuroscience and related fields. Investigators Yang Dan, Tim Blanche, Jack Gallant, and Frederic Theunissen will make several data sets available, each exploring different aspects of sensory coding: (1) cortical slice data acquired in order to examine the effects of complex spike trains in the induction of long-term synaptic modification; (2) recordings of primary visual cortical neurons made during stimulation with complex stimuli, white noise, and natural images; (3) recordings from visual area V4 during stimulation with parametrically varying bars, rings and gratings; (4) recordings from visual areas V1, V2, and V4 during stimulation with a rapid dynamic sequence of gratings; (5) recordings of neurons at three levels of the avian auditory system during stimulation with complex synthetic and natural sounds; and (6) large-scale neuronal recordings from primary visual cortex made with multi-site electrode arrays that allow simultaneous recording from more than a hundred single units at once. It is anticipated that these data will be useful for the study of spatial and temporal neural coding, nonlinear receptive field properties, learning rules, hierarchical processing strategies, and other aspects of the analysis of complex sensory information.

The overarching goal of this project is to discover how language-related information is represented and processed in the human brain. To address this issue we propose to use a novel computational modeling approach, voxel-wise modeling. Voxel-wise modeling draws from the principles of nonlinear system identification, and it provides an efficient method for using complex data sets collected under naturalistic conditions to test multiple hypotheses about language representation. The specific research plan is divided into three aims, each targeted at a different form of language-related information. Aim 1 will reveal how low-level features of speech, such as spectral power, spectral modulation and phonemic structure, are represented across human cortex. Subjects will passively listen to human speech while hemodynamic brain activity is recorded by functional MRI. Voxel-wise modeling will then be used to determine how each point in the brain (i.e., each voxel, or volumetric pixel) is tuned for these various features. Using analogous methods, Aim 2 will reveal how syntactic and semantic features are represented across cortex. Finally, Aim 3 will reveal how language-related information is represented when it is delivered by auditory versus visual modalities. In this case speech and video stimuli will be used. Separate models will be estimated for data recorded during auditory and visual stimulation, and voxel-wise tuning will be compared across modalities. The voxel-wise computational models developed under this proposal will reveal how these various types of language-related information are represented across the cortical surface. These models will also provide clear predictions about how the brain will respond to novel speech stimuli. The results of the proposed research will have broad impacts on clinical problems related to speech perception and production, and they could form the basis of powerful brain decoding device that would enable neurological patients to communicate by thought alone.

? DESCRIPTION (provided by applicant): The human visual system is organized as a parallel, hierarchical network, and successive stages of visual processing appear to represent increasingly complicated aspects of shape-related and semantic information. However, the way that shape-related and semantic information is represented across much of the visual hierarchy is still poorly understood. The primary goal of this proposal is to understand how information about object shape and semantic category is represented explicitly across mid- and high-level visual areas. To address this important issue we propose to undertake a series of human functional MRI (fMRI) studies, using both synthetic and natural movies. Data will be analyzed by means of a powerful voxel-wise modeling (VM) approach that has been developed in my laboratory over the past several years. In Aim 1 we propose to measure human brain activity evoked by synthetic naturalistic movies, and to use VM to evaluate and compare several competing theories of shape representation across the entire visual cortex. In Aim 2 we propose to use VM to evaluate and compare competing theories of semantic representation. In Aim 3 we propose to use machine learning and and VM to discover new aspects of shape and semantic representation. These experiments will provide fundamental new insights about the representation of visual information across visual cortex.

ABSTRACT Natural navigation is an important skill that engages many sensory, motor and cognitive systems. Because aging and degenerative brain disease both diminish the capacity to navigate in the real world, a better understanding of the brain mechanisms mediating navigation will improve diagnosis and monitoring of neurological and neurodegenerative diseases. Neurophysiological studies in animals have led to fundamental insights about the neural mechanisms mediating navigation. However, due to methodological limitations neuroimaging studies of navigation in humans have generally been less compelling than the animal work. We propose to overcome these limitations by using the NexGen 7T MRI scanner recently installed at UC Berkeley to measure brain activity during a naturalistic driving task. Driving is an excellent target for fMRI studies because is a common human navigation task that unfolds across a large and varied landscape, and on a timescale commensurate with fMRI; it engages many navigational brain systems; and it is impacted by aging and neurological diseases. Data will be analyzed by means of an innovative and powerful voxelwise modeling framework developed in PI Gallant's lab over the past 10 years, and validated in many publications. Computational models reflecting 33 different types of navigational features will be fit to the fMRI data separately for each voxel and in each individual subject. Model prediction accuracy and generalization will be cross-validated using separate data sets and subjects reserved for this purpose. The results will be used to test dozens of specific hypotheses about navigation drawn from the theoretical and experimental literature on both rodents and humans. These results will also be used to obtain a detailed functional parcellation of navigational representations in each individual and across the group, and to identify functional networks that represent specific navigation-related features. By combining naturalistic experiments, large-scale computational modeling, multiple hypothesis testing, data-driven functional parcellation and functional network analysis, this research will provide fundamental new information about the human brain mechanisms mediating navigation and their relationship to prior findings from the animal literature.