Lawrence D. Rosenblum, Ph.D. - US grants

ABSTRACT Recently, it has become apparent that a thorough understanding of speech perception must involve an understanding of how audio-visual speech is perceived (Summerfield, 1987). However, relatively little work has been conducted to determine the informational metric for audio-visual speech integration. This research has been designed to determine what this metric is and whether the form of the information for auditory and visual perception is similar. The first series of experiments tests if the perceptual primitives of audio-visual speech perception are articulatory or low-level energy properties. Computer-animated visual displays coupled with discrepant auditory information will be used. The second series of experiments tests the feasibility of modality-independent speech information (Summerfield, 1987). A visual display technique which allows for efficient kinematic analyses will be implemented (Johansson, 1974). The final series of experiments have been designed to determine if kinematic information for speech can specify articulatory dynamics. Visual kinematic properties which are specified to dynamic articulatory states will be determined and then tested in perceptual experiments. These experiments should add to our understanding of speech perception and, specifically, speech (lip) reading. The proposed kinematic analyses and computer animation techniques should help determine the salient kinematic parameters which underlie speech reading.

This research will examine the relationship between speech and speaker perception as it exists across auditory and visual modalities. Recent results have shown that familiarity with a speaker can enhance speech recognition for both auditory speech perception and lipreading. Familiarity with a speaker's voice facilitates auditory speech recognition, and familiarity with a speaker's face facilitates lipreading. These findings challenge traditional theories, which have assumed independence between voice and speech perception, as well as between face recognition and lipreading. Current explanations of speaker-speech facilitation in both modalities have focused on information that is tied to each individual sense. Alternatively, the facilitation could be based on familiarity with a speaker's style of articulation, which is conveyed in both auditory and visual speech information. If the link between speech and speaker properties is based on this modality-neutral articulatory information, then speaker facilitation of speech perception should work across, as well as within, auditory and visual modalities. Three sets of experiments will be conducted to test this hypothesis. The first set will examine whether articulatory information can be used to identify speakers across auditory and visual domains. Experiments will test whether speakers' voices can be matched to their faces based on isolated articulatory information. The second set of experiments will test whether familiarization with a speaker in one modality facilitates recognition of that speaker's speech in the other modality. The final set of experiments will examine the relative influences of switching sensory modality and switching speakers within and between speech utterances. The results of this research should be illuminating about theories of speech and face perception, as well as general issues of multimodal integration. The research will address issues relevant to individuals with hearing impairments, as well as aphasic, prosopagnosic, and phonagnosic patients.

[unreadable] DESCRIPTION (provided by applicant): A thorough understanding of language must involve examination of how language is used in its most common setting: between interlocutors. During live conversations, participants usually have access to visual as well as auditory speech and speaker information. While much research has examined how the auditory speech signal supports live conversation, relatively little research has addressed how visual speech (lipread) information aids conversation. This fact is unfortunate given what is now known about the ubiquity. automaticity, and neurophysiological primacy of visual speech perception. The proposed research has been designed to broaden our understanding of how visual speech is used in conversational settings. One way in which shared understanding emerges from conversation is through the cross-speaker language alignment known to exist at all levels of dialogue. Interlocutors align to (partially imitate) each other's speech so as to converge towards a common tempo, intonation, and the more microscopic dimensions of voice onset time and vowel spectra. The proposed research will examine how visual speech information influences speech alignment. This research will also examine two important theoretical issues in the contemporary speech literature, namely: 1) the role of talker information in phonetic perception; and 2) the role of external perceptual information in controlling speech production. The experiments will test whether visual speech information influences speech production alignment in the context of both a simple word shadowing task and a two-participant interactive task. Manipulations will be incorporated to determine whether the talker information that influences production responses is available cross-modally. Another set of experiments will test whether visual speech information can modulate speech production so as to facilitate speech production response times. The results of this research should have important implications for theories of speech and speaker perception, as well as our understanding of multimodal integration, imitation, and the relationship between perception and action. The experiments should also add to our knowledge of the salient information for (Spreading, face recognition, and voice identification. The research will address issues relevant to individuals with hearing impairments, as well as aphasic, prosopagnosic, and phonagnosic patients. [unreadable] [unreadable] [unreadable]

When listening to someone in a noisy environment, you can better understand what they say when you look at their face while they talk. Providing audiovisual information about speech, not just audio information alone, has also been shown to help infants develop language. In some cases, using audiovisual information to train speech perception helps adults learn a second language, helps aphasic patients produce more fluent speech, and improves perception of heard speech by listeners with hearing impairments. This last group is of particular interest because roughly a third of U.S. adults have some degree of hearing impairment.

While this audiovisual training advantage is well established, it is unclear how exactly the effect works. Research will be conducted to determine the principles behind the effect, how and why it works, and in what circumstances. The investigators also explore why audiovisual training typically does not help hearing-impaired individuals who received cochlear implants after early childhood. One possibility is that their years of experience before receiving the cochlear implant required a heavy reliance on lip reading, leaving them over-reliant on information from the talker during audiovisual training.

The specific issues examined are rooted in theories of multisensory learning. These issues include: a) whether simultaneous auditory and visual speech is necessary for the training advantage to occur; b) to what degree the advantage generalizes across different talkers for a given listener; and c) whether the use of other sensory channels, in this case audio and touch, results in a similar training advantage. Parallel experiments will be carried out with both normal-hearing individuals and individuals who received cochlear implants after early childhood. The research should not only improve speech perception training regimes for individuals in this clinical population but should also improve our understanding of typical speech and talker perception. More generally, the research should inform our understanding of perceptual learning and plasticity, as well as multisensory integration.