Louis D. Braida - US grants

This research is directed toward improved signal-processing schemes to aid people with sensorineural hearing impairments. We intend either to develop improved schemes or to obtain a fundamental understanding of why such schemes cannot be developed. The proposed work includes study of linear amplification, amplitude compression, and frequency lowering. Also, attempts will be made to determine and understand the effects of variations in speech production on speech reception by impaired listeners. The research on linear amplification is concerned with modelling the dependence of speech-reception performance on the speech materials, the background interference, the listener, and the linear amplification system. Initial work on this project involves an attempt to apply Articulation Theory to speech reception by impaired listeners. The research on amplitude compression, directed towards listeners with reduced dynamic range, involves further study of multiband syllabic compression, as well as study of automatic volume control and limiting. The research on frequency lowering, directed towards listeners with poor hearing at high frequencies, continues to focus on pitch-invariant, nonuniform lowering. In both areas, the proposed work involves further exploration of the effects of various system parameters on speech-reception performance, testing of a wider variety of subjects, and attempts to determine the underlying causes of the results obtained. The research on the effects of variations in speech production is motivated by our belief that there exist variations, both intersubject and intrasubject, that lead to substantially improved speech reception, and that an understanding of these variations and their effects will provide useful background for the development of improved signal-processing schemes. The proposed work in this area involves the development of speech materials uttered by different speakers under a variety of speaking conditions, the measurement of speech reception by impaired listeners using these materials under a variety of listening conditions, and the attempt to correlate the speech-reception results with properties of the acoustic waveforms.

This research is directed toward improved speech reception for the hearing impaired. Attempts to advance basic understanding involve study of limitations imposed by characteristics of the speech signal and of the impairment, and the development of a new model of speech intelligibility that exploits ideas used in automatic speech recognition. Research on the speech signal focuses on understanding the effects of environmental disturbances (noise and reverberation), intra-speaker and inter-speaker variability of utterances, and efforts to speak clearly. Research on impairments focuses on developing and testing of techniques for simulating impairments with normals and estimating the speech reception capacity of impaired auditory systems through tests with artificial speech codes. The envisioned model of speech intelligibility includes models of perceptual processing and vector quantization, and makes use of a mutual information metric to predict intelligibility. Attempts to develop improved signal processing techniques for use in hearing aids involve study of adaptive linear filtering, amplitude compression, and frequency lowering. Also considered is an extension of articulation theory to account for speechreading and the use of this extended theory to develop effective supplements to speechreading. Attempts to apply automatic speech recognition to the development of speechreading supplements involves study of manual cued speech and the Autocuer, determination of the extent to which modern speech recognition algorithms can disambiguate speechreading, and study of auditory and visual displays for presenting speechreading supplements based on the output of automatic recognition systems to impaired listeners.

This research is directed toward improving speech reception for the severely hearing impaired who rely on speechreading for communication. Attempts to advance basic understanding involve study of speechreading and the effects of auditory and visual supplements on audiovisual speech reception. Attempts to develop supplements for speechreaders involve extracting effective cues from acoustic speech and displaying them to individuals with severe hearing impairments. Attempts to develop mathematical models of audiovisual integration focus on quantifying low well supplementary signals are integrated with speechreading in the reception of speech segments, suprasegmental characteristics, and sentences. Model predictions are compared to measured speech reception for a variety of listeners and presentation conditions, including degraded auditory and visual reception, to measure the efficiency of audiovisual integration. Attempts to develop signal processing techniques to derive speechreading supplements from acoustic speech will be concerned with simplified signals that can be readily matched to residual hearing. The signals must be accurately derived from speech degraded by interference and reverberation. Attempts to determine the factors that limit the effectiveness of such signals will provide foundation for realistic evaluations of promising simplified signals as speechreading supplements. Attempts to apply automatic speech recognition to the development of speechreading supplements involve the study of Manual Cued Speech, determination of the extent to which modern speech recognition algorithms can produce speechreading cues automatically, and study of visual displays for presenting speechreading supplements based on the output of automatic recognition systems to impaired listeners.

DESCRIPTION (provided by applicant): This research is directed toward improved speech reception for users of hearing aids and cochlear implants through a program of research on models of speech intelligibility. The goal is to develop and experimentally evaluate a robust physical metric that predicts intelligibility scores for a variety of alterations of the speech signal for both hearing-impaired (HI) listeners and Cochlear Implant (Cl) users. This metric will be an adaptation of the speech-based Speech Transmission Index (sSTI) that computes a novel metric from the speech signals directly and the characteristics of individual listeners, and deals with a wide class of nonlinear distortions that are not adequately modeled by traditional STI methods. We have 4 aims. The first aim is to measure speech reception in 3 classes of listeners: HI listeners, Cl users, and normal-hearing subjects listening through a channel-vocoder simulation of cochlear-implant sound processing for 4 types of alterations of speech (acoustic degradations arising from noise and reverberation, band-pass filtering, amplitude compression, and noise-reduction algorithms). The second aim is to characterize the relevant basic psychoacoustic abilities of individual HI listeners and Cl users (in terms of basic sensitivity, dynamic range, spectral resolution, and temporal resolution) and their ability to integrate cues across different filtered bands of speech. The third aim is to develop sSTI-based metrics of speech intelligibility and apply them to the stimuli used to test HI and Cl listeners. The metrics will incorporate the individual listener characteristics, both psychoacoustic abilities and facility at integrating cues across frequency. The fourth aim is to evaluate these metrics by comparing metric predictions for a variety of listeners and speech processing conditions to the empirical data we have obtained on measures of speech reception. At the end of this project, we will have a single metric that is applicable to both HI listeners and Cl users (and by extension to normal-hearing listeners as well). This metric will predict the effects of noise, reverberation, filtering, amplitude compression, and noise reduction algorithms on speech intelligibility and will have a number of applications in research and clinical settings.