Robert E. Remez - US grants

A series of experiments is proposed to investigate sensory and perceptual processes in speech communication. These experiments will attempt to clarify the mechanisms of perceptual organization of the speech signal, which operate early in the processing of acoustic stimuli. The processes that organize phonetic signals define coherent patterns of stimulus elements, establishing an auditory representation appropriate for phonetic recognition. Psychophysical research on auditory organization provides one source of hypotheses about the treatment of speech stimuli. In this vein, current concepts of frequency analysis will be applied to the identification of sentence intonation, with specific reference to the concept of the dominance region in pitch perception. The proposed research also seeks to extend the analysis of auditory streaming to include the organization of speech signals. The projects examine the integration of concurrent variations in formant frequency; the basis for perceptual continuity when stimulus elements in the speech stream are physically discontinuous; and the contribution of the perceiver's knowledge of natural spectrotemporal signal changes in these processes. The organizing effects of lexical structure will also be investigated. These studies will identify the means by which the perceptual integrity of the speech signal is established and maintained, thereby contributing to our understanding of speech perception by normal human adults.

A series of experiments is proposed to investigate sensory and perceptual processes in speech communication. These experiments will attempt to clarify the mechanisms of perceptual organization of the speech signal, which operate early in the processing of acoustic stimuli. Such processes organize phonetic signals by defining coherent patterns of stimulus elements, establishing an auditory representation appropriate for phonetic recognition. Two sources of hypotheses about organizational mechanisms active in the perception of speech will be investigated. Psychoacoustic research on auditory organization will motivate a number of empirical analyses of the role of stream formation in the perception of speech signals. Perceptual and acoustical studies of vocally produced sounds will also be used to consider organizational mechanisms that may be tailored specifically to speech signals. The four projects that are planned will investigate the contingency of phonetic perception on the range of signal frequency variation; the relative disposition to organize sinewave replicas of speech auditorily or phonetically; the growth of attention to time-verying spectrum properties; and the perception of spontaneously produced speech. These studies will identify the means by which the perceptual integrity of the speech signal is established and maintained, thereby contributing to our understanding of speech perception by normal human adults.

Scientists and engineers have studied speech to understand why this form of communication is so effective. They have also sought to create speaking and listening devices that approach the accuracy and ease of everyday communication, with modest success. The research problem has been easy to define: English is composed of more than 100,000 words created from over 16,000 different syllables and syllables are composed from a small inventory of several dozen consonants and vowels. Automatic speech recognition would be remarkably easy if these linguistic properties - words, syllables, consonants, vowels -produced uniformity in the sounds that talkers actually make. In fact, each utterance is also physically unique, whether in its sound pattern or in the visible movements of the speaker's face. Different vocal anatomy in men, women and children causes complex variation in sound production even when the linguistic message is the same. Moreover, aspects of a talker's productions may express the dialect and speaking style of their family and linguistic community. Human listeners readily attend to the acoustic hints of these individual and social markers while also listening for the message. This project will examine how these different sources of perceptual information for speech are organized, how they are integrated over time, and how they allow perceptual tuning to the speech of individual talkers. Ultimately, a more complete account of the perception of speech can lead to improvement in recognition technology and to the creation of assistive devices.


Three experimental projects will be performed: 1) to estimate the temporal dynamics of auditory sensory integration; 2) to determine the dimensions of exposure-based perceptual tuning to the characteristics of individual talkers; and, 3) to describe and model the intrinsic differences in auditory and visual temporal sensitivity and persistence that affect audiovisual speech perception. In each instance, the perceptual sensitivity to linguistic properties, talker characteristics, and language general features of spoken language will be assayed using discriminating and robust measures of auditory and audiovisual resolution. The studies explore the versatility of perceptual faculties applied to speech and provide an opportunity to identify the principles underlying the remarkably robust perceptual abilities that support and sustain communication. The overall goal is a formal and functional characterization of the cognitive resources that insure the perceptual stability of spoken communication in natural environments, whether the source of speech is visible or not, whether the talker is familiar or not, and whether the quality of the sensory samples of speech is natural or not.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.