Frederic Theunissen - US grants

DESCRIPTION (provided by applicant): Our auditory system analyzes complex sound waveforms in an amazing diversity of ways. First, it extracts features that give us the percepts of rhythm, timbre and pitch. Simultaneously those features are combined and compared to stored memories to produce higher-order percepts such as the meaning of speech, the speaker's identity and her emotional state. Second, these auditory tasks are often performed with environmental noise or other interfering acoustical signals in the background. Finally, our auditory system needs also to process the sound of our own voice to guide our vocalizations. We propose to study the auditory system of songbirds as a model system to understand the neural computations of circuitry underlying these diverse abilities. In previous work, we have characterized responses in the avian auditory midbrain, thalamus and in the primary and secondary auditory cortex. We have shown that auditory neurons are specialized to represent natural sounds and that we can explain this specialization from their tuning properties. We also found evidence for parallel functional processing streams: auditory neurons in the midbrain and thalamus fall into different functional types based on how they decompose sound into features that are crucial for different auditory percepts. Our major goals for this project are 1) to relate the functional properties of neurons to the anatomy and microcircuitry of the auditory cortex, 2) to begin to unravel the cellular computations that lead to the observed functional specialization and 3) to investigate the computations in the primary and secondary auditory cortex that could allow the system to process signals in noise. To achieve these goals we will record from single neurons in the anesthetized preparation, both with multi-electrode arrays for extracellular recordings or with glass electrodes for intracellular recordings and immunohistochemical work. We will also record neural activity in awake behaving birds using a miniaturized electrode drive. The birds will be placed in situations that elicit them to actively communicate. In all our experiments, we will analyze the neurons'tuning and selectivity using state-of-the-art techniques from systems analysis and information theory. Our studies will elucidate the roles of different circuits within auditory cortex in processing complex sounds such as speech and music. This knowledge will be essential to understand how dysfunctional auditory processing in certain mental disorders affects speech recognition and consequently other cognitive skills. Our work could also be instrumental in the development of novel signal processing methods for auditory neural prosthetics. PUBLIC HEALTH RELEVANCE: The purpose of this research is to discover how neural circuits in the auditory thalamus and cortex process complex sounds so that we perceive in them different acoustic qualities, such as pitch and timbre, which, in the case of communication sounds, contribute to a higher-level perception of content that has behavioral meaning, as in the understanding of speech. This research will be useful for designing the next generation of hearing aids and cochlear implants, and will allow us to understand the causes of some learning disabilities and mental disorders that involve high-level auditory processing including deficits in speech comprehension and other cognitive abilities.

DESCRIPTION (provided by applicant): Understanding sensory systems relies on characterizing the stimulus-response properties of neurons at each stage of processing. This characterization can then be used to investigate the neural mechanisms underlying sensory maps and to derive computational models of sensory processing. Such knowledge will enable us to better understand the computations performed by the human brain in general and in particular will provide insights on how to develop better sensory prosthetics. We propose to develop a novel suite of quantitative methods for objectively characterizing the nonlinear responses of sensory neurons. A unique feature of our methods is that they can be used with complex, naturalistic stimuli as well as with conventional simple stimuli commonly used in sensory neurophysiology. The powerful combination of nonlinear analysis and complex stimuli potentially enables us to characterize sensory neurons even at relatively high levels of sensory processing. Much of our proposal focuses on developing the quantitative computational tools necessary for our analyses. First, we propose to develop algorithms for estimating nonlinear receptive field profiles of sensory neurons from responses to arbitrary stimuli. Second, we propose to develop tools for evaluating the quality of the resulting receptive field models. Third, we will develop analysis tools that will facilitate the biological interpretation of the derived models. Finally, we propose to develop a comprehensive software package that will include both linear and nonlinear estimation techniques as well as the evaluation and analysis tools. The package will consist of a stand alone documented function library for the most experienced users, a turn-key package with an integrated graphical user interface for general users and extensive online documentation. The tools and analyses will be validated using computational models and data collected in our laboratories and in other laboratories that have agreed to aid us in software testing and evaluation.

DESCRIPTION (provided by applicant): Using as a starting point the postulate that sensory systems have evolved to perform optimal transformations on behaviorally relevant or natural stimuli, we will use systems analysis methods and information theoretic principles to develop a theory of auditory processing. The goals of our theory will be to predict the stimulus-response transformations that are found at different stages of auditory processing. First, we will obtain theoretical predictions for the distribution of linear receptive fields by jointly maximizing signal to noise ratio and entropy in the output of the ensemble of filters when presented with natural sounds. Second, we will derive non-linear stimulus-response transformations that can be obtained with biologically plausible networks and that will minimize the mutual information across neurons. These neural networks will perform a form of independent component analysis, in which the resulting operation is to extract independent acoustical features in natural sounds. We will also develop novel methods to estimate the information transmitted by single neurons and ensembles of neurons in songbirds. The goodness of fit of the theoretical models will be assessed in two steps. First we will compare the theoretical stimulus-response functions with the functions of the same order that will be obtained directly from the neural data. Second, we will evaluate how well the stimulus-response functions describe the actual neural transformation by comparing predicted responses and predicted information rates with actual responses and measured information rates. Our analysis will give us insight on how complex natural sounds are processed in the auditory system of animals and humans. Understanding how biological systems process natural sounds will be instrumental in the development of novel algorithms in engineering applications for sound compression, speech recognition and sound pre-processing for hearing aids and auditory prosthetics.

Proposal No: 0749056
PI: Yang Dan

Award Abstract:

This award supports the preparation and sharing of computational neuroscience data as part of an exploratory activity aimed at catalyzing rapid and innovative advances in computational neuroscience and related fields. Investigators Yang Dan, Tim Blanche, Jack Gallant, and Frederic Theunissen will make several data sets available, each exploring different aspects of sensory coding: (1) cortical slice data acquired in order to examine the effects of complex spike trains in the induction of long-term synaptic modification; (2) recordings of primary visual cortical neurons made during stimulation with complex stimuli, white noise, and natural images; (3) recordings from visual area V4 during stimulation with parametrically varying bars, rings and gratings; (4) recordings from visual areas V1, V2, and V4 during stimulation with a rapid dynamic sequence of gratings; (5) recordings of neurons at three levels of the avian auditory system during stimulation with complex synthetic and natural sounds; and (6) large-scale neuronal recordings from primary visual cortex made with multi-site electrode arrays that allow simultaneous recording from more than a hundred single units at once. It is anticipated that these data will be useful for the study of spatial and temporal neural coding, nonlinear receptive field properties, learning rules, hierarchical processing strategies, and other aspects of the analysis of complex sensory information.

DESCRIPTION (provided by applicant): Using as a starting point the postulate that sensory systems have evolved to perform optimal transformations on behaviorally relevant or natural stimuli, we are using signal analysis methods and information theoretic principles to develop a theory of auditory processing. The purpose of our theory is not just to describe but to understand the neural representation of acoustic communication signals, including speech and music. First, we plan on analyzing the statistics of natural sounds and of speech, music and birdsong in particular. We propose to search for theoretical representations of sounds based on principles of statistical independence and sparse representation. Our derived representations will also attempt to maximize differences between acoustic features that meditate the qualitatively different acoustical percepts of rhythm, timbre and pitch. Second, we will test the validity of these theoretically derived representations in psychophysical experiments in humans, and behavioral experiments in songbirds. These experiments will test the effect on perception of systematically removing acoustic features along the particular dimensions that were derived in the statistical analysis. Third, we will develop information theoretic tools that will allow us to estimate the amount of redundancy in a neuronal ensemble response. These measures will be used to quantify how the neural representation changes as one ascends the auditory processing stream and to test whether the neural representation is becoming more sparse and independent as we theorized. Finally, we will record the neural responses in primary and secondary auditory areas in songbirds to playback of song and filtered song. The data from these neurophysiological experiments will be used to: 1) test the utility of the statistically derived representations to predict responses of single auditory neurons, 2) correlate neural responses and behavioral responses, 3) assess the nature of non-linearities in the response, and 4) test the assumptions of independence at the ensemble level. Our studies will give us insight on how speech, music and other complex sounds are processed by the auditory system. These studies could be instrumental in the development of novel methods for sound processing for hearing aids and auditory neural prosthetics, as well as diagnostic tools for classifying language and learning disorders. PUBLIC HEALTH RELEVANCE: The purpose of this research is to improve understanding of how the human brain represents complex sounds, speech and music in particular. This basic research has direct applications in bio-medical sciences in the development of better hearing aids or cochlear implants for better speech intelligibility and music appreciation. In the long term, the research might also help in understanding the cause of certain types of learning disabilities that involve poor speech comprehension or production.

DESCRIPTION (provided by applicant): This is a proposal to support a biennial international meeting on the topic of Sensory Coding and the Natural Environment. Sensory neuroscience is currently undergoing a rapid explosion in technologies that enable experimentation that is no longer constrained to the use of immobilized animals subjected to highly simplified stimuli. Large numbers of implantable electrodes able to transmit data about neural activity, advances in imaging techniques, the possibility to manipulate neuronal activity using light impulses while animals are awake and behaving, and the computational resources and techniques to simulate dynamic and responsive environments in the laboratory are only a few examples of the emerging sophistication in experimental methods. These methods permit direct observation and manipulation of the brain as it performs natural tasks in its natural environment. On one hand, theoretical and experimental studies have shown that sensory systems are able to exploit the complex correlations found in natural sounds and scenes to optimize their processing capacity. On the other hand, the complexity of the "real world" poses particular challenges, such as the sound source separation in the cocktail party problem. It is therefore critical to develop appropriate methods for handling input/output relationships of this complexity and to develop theoretical concepts that frame predictions about coding principles. These methodological and theoretical advances are key to the design of better hearing aids, sensory prosthetics or other technologies (such as the use of computer enhanced virtual environments) to address sensory or communicative disorders. This meeting therefore aims to bring together an interdisciplinary group of researchers with expertise in systems and cognitive neuroscience, perceptual psychology, statistics, signal processing, computer science and sensory aids to discuss the forefronts of research and emerging common principles and methods in sensory neuroscience in the context of natural environments and behavior. PUBLIC HEALTH RELEVANCE: Characterizing natural scene statistics and the complex, adaptive neural responses they elicit is essential for shedding light on neural information processing strategies, as well as for advancing the development of neural prostheses capable of transforming sounds and images encountered during natural behavior into a format interpretable by the brain. The goal of this project is to help fund an interdisciplinary meeting to discuss the latest findings on these topics.

The overarching goal of this project is to discover how language-related information is represented and processed in the human brain. To address this issue we propose to use a novel computational modeling approach, voxel-wise modeling. Voxel-wise modeling draws from the principles of nonlinear system identification, and it provides an efficient method for using complex data sets collected under naturalistic conditions to test multiple hypotheses about language representation. The specific research plan is divided into three aims, each targeted at a different form of language-related information. Aim 1 will reveal how low-level features of speech, such as spectral power, spectral modulation and phonemic structure, are represented across human cortex. Subjects will passively listen to human speech while hemodynamic brain activity is recorded by functional MRI. Voxel-wise modeling will then be used to determine how each point in the brain (i.e., each voxel, or volumetric pixel) is tuned for these various features. Using analogous methods, Aim 2 will reveal how syntactic and semantic features are represented across cortex. Finally, Aim 3 will reveal how language-related information is represented when it is delivered by auditory versus visual modalities. In this case speech and video stimuli will be used. Separate models will be estimated for data recorded during auditory and visual stimulation, and voxel-wise tuning will be compared across modalities. The voxel-wise computational models developed under this proposal will reveal how these various types of language-related information are represented across the cortical surface. These models will also provide clear predictions about how the brain will respond to novel speech stimuli. The results of the proposed research will have broad impacts on clinical problems related to speech perception and production, and they could form the basis of powerful brain decoding device that would enable neurological patients to communicate by thought alone.

Human speech and animal communication require both the extraction of meaning from sound and the processing of one's own voice to guide the production of these vocalizations. These processes require non-trivial computations that have challenged linguists and engineers but that are performed effortlessly by our brains. To understand what are the neural computations performed to decode the behavioral meaning and vocal gestures of communication signals, this study will examine how the auditory cortex of a songbird processes the complete vocal repertoire of its own species.

The Theunissen Lab acquired a unique database of all the vocalizations emitted by adult and juvenile, and both male and female zebra finches. This database contains the complete repertoire with multiple exemplars of each vocalization type for many individuals. Because the behavioral context of each communication sound was carefully recorded, these sounds are classified in meaning categories. This database will thus enable the detailed investigation of how the auditory system extract meaning from vocalizations, while controlling for variability of production within vocalization type as well as between individuals.

The approach of this project consists in obtaining neural responses to these communication sounds using advanced neurophysiological recording techniques, and then investigating the neural computations by finding the statistics models that best predict these responses. Multi-electrode arrays will be used to record the simultaneous neural activity of large sets of single neurons in the primary and secondary auditory areas. The response of these neurons will then be fitted using statistical models that incorporate increasing levels of abstraction: from elementary sound features, to vocal gestures and semantic labels. The representation in terms of vocal gestures will be obtained from a reduced physical model of the avian vocal organ. This analysis will not only point out the brain regions that are involved in semantic processing but also the nature of the hierarchical computations that lead to these higher-level representations. The research will also investigate the link between perception and production by directly assessing the role of a motor-based representation of sounds in high-level auditory areas.

By combining ethological, neurophysiological and computational studies of acoustic communication in a songbird, the project will establish an appropriate animal model system to elucidate how the auditory cortex extracts and categorizes sound features in order to link sound to meaning. Given the similarities in the anatomy and physiology of the auditory system across vertebrates and the common signal processing problems shared in all vocal communications, this study can also contribute significantly to the neurophysiological understanding of neural mechanisms underlying speech perception.

This award is being co-funded by NSF's Office of the Director, International Science and Engineering. A companion project is being funded by the French National Research Agency (ANR).

Vocal communication relies on the production and perception of stereotyped signals that carry particular messages. Bioacousticians studying animal communication and linguists studying human speech have used a unique simple classification scheme for these information-bearing sound elements: phonemes in speech and call categories in animal calls. Although appealing from a computational perspective, classification of communication sounds into a fixed number of phonemes or call types fails to capture all of the complexities of the vocal signaling used by humans or animals. At the same time, this core view has guided past neuroscientific research, including ours: auditory neuroscientists and neurolinguists have searched with mixed success for high-level auditory neurons or regions that are selective for call types or phonemes and invariant to variations of sounds within these groups. One of the sobering lessons of this past research is that we have yet to find a cortical map representing phonemes or call types. In addition, the non-linear transformations yielding invariant and selective responses for certain phonemes/call types remain to be described. For this project, we propose to explore a broader hypothesis of auditory signaling and perception and explore its neural correlate. Our hypothesis is that signaling is organized in an information hierarchy that is best represented as a tree classification scheme. This hierarchical organization of information is found in the acoustics, in the behavioral responses and in the neural representations and computations performed by the auditory system and association areas. We propose to test this hypothesis and reveal the neural representations of communication signals using songbirds as a model system. In this collaborative proposal, we will 1) develop a novel wireless device that will combine array neural recordings, physiological measurements and acoustical recordings, 2) study the hierarchical organization of a complete repertoire as revealed by the acoustical structure of calls, 3) measure the physiological and behavioral responses to these communication sounds, 4) measure the neural representation for these communication sounds and 5) model this communication system using our hierarchical information hypothesis as a guiding principle. This analysis will elucidate not only the nature of the meaning in communication signals but also how the sound to meaning transformations are performed by the brain.

Project Summary/Abstract To interpret the vocalizations used in communication, such as human speech, animals and humans must perform a range of auditory tasks: the detection and localization of sounds, the perception of pitch and timbre, and the parsing and categorization of the information bearing sound features that is required for the interpretation of communication calls. Auditory neuroscientists have obtained a relatively good model of how complex sounds are represented in the primary auditory cortex primarily in terms of their spectro-temporal features. We also know that a network of higher-level auditory and associative cortical areas is involved in processing speech in humans and communication calls in animals. However, the neural circuits and the corresponding non-linear transformations that occur between primary auditory cortical areas and cortical regions that categorize communication sounds in terms of their meaning remains unknown. We are developing the avian model system to bridge this gap. Songbirds have a large repertoire of communication sounds that are used in distinct behavioral contexts. By combining behavioral and neurophysiological experiments, we will investigate how calls are categorized into call-types (semantics). We will also investigate the neural representation for learned categories that correspond to different vocalizers (voice). Using state-of-the-art computational approaches, we will decipher the sequence of non-linear processing steps occurring both at the level of single neurons and neuronal ensembles that perform these sound- to-meaning transformations. Our studies will elucidate the roles of different circuits within auditory cortex for processing semantics and voice. This knowledge will be essential to understand how dysfunctional auditory processing in certain mental disorders affects speech recognition and consequently other cognitive skills. Our work could also be instrumental in the development of novel signal processing methods for auditory neural prosthetics or hearing aids.