Peter C. Gordon - US grants

The goal of this project is to identify factors that contribute to the ease with which the same information can be effectively presented by a computer system in either written or spoken language. Advances in computer technology lead to the prospect that in the near future linguistic messages could be automatically converted from text to speech or from speech to text. This project will test the efficacy of strategies both for converting written language to spoken language, using text-to-speech synthesis systems, and for converting spoken language into written language. It will also assess the degree to which user control of spoken-language presentation can enhance understanding of computer-based spoken messages by providing the auditory equivalent of rereading and scanning ahead. Having the same message available in either spoken or written language would greatly enhance access to information technology when reading is impossible (because of the situation or because an individual has vision problems or limited literacy) or when understanding spoken language is impossible (because a situation requires quiet or because an individual has hearing deficits). The results of the project will help to enhance the everyday use of computer technology, especially for individuals with impairments of hearing or vision.

This project will examine the cognitive processes that are engaged by the probe-word tasks that have commonly been used to study language comprehension, focusing on the processes involved in the comprehension of referring expressions during the reading of narrative texts. It will do so in order to determine the extent to which the memory processes measured by probe tasks are those that are usually involved in language comprehension and the extent to which they arise from strategic task adaptations. The project seeks to gain an understanding of the mechanisms by which referential expressions are interpreted and by which patterns of coreferential expressions contribute to the meaning and coherence of sentences within a discourse Empirical work will use probe word recognition tasks. These tasks involve presenting subjects with a probe word and asking them to make a speeded response indicating whether or not that word has occurred in the passage that they are reading (or hearing). Use of the task is based on the assumption that response times to the probe word reflect the accessibility of discourse referents in the memory representations that are being constructed as part of language comprehension. The research using the task aims to test the validity of that assumption in specific types of studies on coreference. The probe word recognition tasks will use rapid serial visual presentation (RSVP) with the sentence or passage being presented at the center of a computer screen The exploration of the probe word task will focus on two major issues: How do the mechanisms for coreferential interpretation of reduced expressions (specifically pronouns) and complete referring expressions (specifically proper names) differ and what is the relative ease of interpreting these two forms of expressions in a coreferential manner? Probe word recognition experiments will examine the extent to which results showing that repeated name coreference increases memory accessibility more than pronominal coreference may be due to strategic adaptations to the probe word recognition task. How do the sequential and syntactic structure of language influence mental representation in a model of discourse and how does this mental representation influence the interpretation of referring expressions in linguistic input? Probe-word recognition experiments will examine the extent to which evidence on the importance of sequential organization (particularly first mention) on memory accessibility are due to strategic task adaptations.

With National Science Foundation support, Drs. Peter Gordon and Randall Hendrick will conduct three years of linguistic research on the functions of human memory during language comprehension. Their focus is working memory, which is the memory that keeps track of information as people actively engage in complex tasks. Language comprehension includes complex tasks like identifying and interpreting parts of sentences. Such tasks require that intermediate representations of sentence parts be held in working memory and accessed during comprehension. The project tests the hypothesis that working memory's constraints in language, and other cognitive domains, reflect its susceptibility to interference during encoding, storage, and retrieval. This interference arises from the similarity of the items being processed. This research uses three complementary methods: eye tracking during reading, analyses of patterns in large corpora of printed language, and memory load tasks during language comprehension. The broader aims of the project are pursued by focusing each method on two central issues: How is language processing affected by the proximity of similar expressions, and what are the psychologically salient dimensions of similarity that cause interference?

The significance of the project is that it will advance knowledge of the relationship between language and other aspects of cognition, particularly memory. Basic scientific research on the organization of the cognitive processes involved in crucial facets of mental life will be of benefit for later development of more applied goals. These include the creation of effective information technology, the teaching of reading, and the diagnosis and treatment of language impairment.

DESCRIPTION (provided by applicant): The understanding of referential and coreferential expressions is an essential part of the mental construction of a discourse model that embodies the meaning of a sentence or a series of sentences. Correct understanding of (co) reference is essential both to integrating and differentiating meaning in a discourse model; it is the basis by which the characteristics and actions associated with different expressions that refer to the same entity are integrated together and by which they are simultaneously differentiated from the characteristics and actions associated with different entities mentioned in the text. In order to do their work, the mechanisms of coreferential processing must interact with basic memory processes that are involved in the processing of words. Research on lexical memory processes has for a long time used repetition priming and semantic priming in order to understand basic episodic and semantic characteristics of word processing; this work has provided the basis for many commonly accepted mechanisms of cognitive processing. However, the correct interpretation of reference and coreference places very different demands on information processing from those that arise in the processing isolated words or lists of words. The current project examines how language comprehension uses, and in some cases overrides, basic mechanisms of lexical processing. The project addresses five specific aims: (1) to investigate memory integration of coreferential expressions, (2) to compare memory integration in structured language and word lists, (3) to investigate memory integration of coreferential expressions consisting of category terms, (4) to investigate the nature of linguistic focus and its relation to other kinds of attention, and (5) to investigate the role of the right hemisphere in coreferential processing. Throughout the proposed research, the combined use of eye tracking and ERPs during reading will provide complementary measures of the cognitive and neural processes involved in language comprehension. Because language is an essential cognitive process, elucidating basic language mechanisms remains a high priority in the effort to understand, diagnose and treat developmental and acquired language impairments.

In what ways are human language and formal, computer-based languages similar? In what ways are they different? How can understanding of human languages lead to more effective computer languages and how can the explicit formalization offered by computer languages be used to increase understanding of how humans use language? These questions are central to efforts to understand language as a cognitive activity, an activity that plays a critical role in communicating, transforming, storing and retrieving information. With NSF funding, language researchers at the University of North Carolina at Chapel Hill will bring together the world's leading researchers to address these basic questions in a special sentence of the 21st Annual CUNY Conference on Human Sentence Processing in Chapel Hill, North Carolina in March 2008. In this special session, the latest developments in formal models of language processing, using techniques developed in computer science, neural network research and statistics will be applied to human language use. In addition, the special session will address how the sharing of formal tools (e.g., model simulations, datasets, and statistical procedures) could have a transformative effect on the study of human sentence processing by facilitating wider use of formal methods and by providing a way in which this research community could more effectively exploit advances in modeling and statistical analysis that have been developed in other scientific domains.

The Annual CUNY Conference on Human Sentence Processing is the most prominent meeting of high-level language-processing researchers in the world. In addition to directly supporting the special session on formal models, NSF funding will help support the participation of beginning investigators, including starting faculty, post-doctoral researchers and graduate students, and so will facilitate entry of new generations of researchers into the language sciences.

[unreadable] DESCRIPTION (provided by applicant): The goal of this project is to increase understanding of adaptation in human auditory cortex after peripheral hearing loss in adults. Two types of cortical adaptation have been studied for hearing: expansion of cortical representation for the frequency that is at the edge of a sharply-sloping hearing loss and a reduction in the usual pattern of cortical lateralization (more contralateral than ipsilateral activity) in cases of unilateral deafness. We have used functional magnetic resonance imaging (fMRI) and psychoacoustic tasks to investigate the relationship between cortical adaptation and perceptual ability in two individuals with a sloping hearing loss in one ear and normal hearing in the other ear. Both individuals showed reduced lateralization of activity in auditory cortex in response to sounds heard in their normal ear as compared to their impaired ear. Reduced lateralization after stimulation of the normal ear was attributable to an increase in ipsilateral cortical activation and was observed both at frequencies where hearing was absent in the impaired ear and also where it was fully preserved as indicated by audiometric criteria. Evidence of perceptual changes in the patients that we studied was found in two binaural listening tasks where a signal was presented in one ear and a distractor sound in the other. Performance on signals presented to the impaired ear was more susceptible to interference from the normal ear than vice versa even at frequencies where audiometric sensitivity in the two ears was equivalent. Together, the fMRI results and psychoacoustic results provide a compelling parallel between cortical adaptation and perceptual function in adult humans. This project uses fMRI to determine further the nature and extent of long-term neural adaptation in individuals with unilateral hearing impairment, and psychoacoustic experiments to determine the nature of the perceptual changes that are associated with cortical adaptation. Increased understanding of long-term cortical adaptation to sensory loss has the potential to provide insight into how processes of learning are constrained both by patterns of neural connectivity created during development and by the limitations on neural growth in mature animals. Increased understanding of cortical adaptation also has the potential to shed light on how higher-level neural processes influence the benefits and limitations of interventions that restore (or partially restore) lost sensory input. [unreadable] [unreadable] [unreadable]

This project investigates how the mind coordinates information in human memory with the presentation of novel information provided during language comprehension. The focus is on the dynamic construction of a mental representation of discourse through the interpretation of referential expressions, typically pronouns. The work investigates the manner in which this process is mediated by representational aspects of episodic and semantic memory, and will integrate knowledge about memory and language processing to explain how the ambiguity, frequency, and semantic content of noun phrases affect a critical aspect of language comprehension, namely the interpretation of expressions that refer to entities mentioned previously.

The primary method used in the research is to measure the eye movements of adults as they read short passages; an additional method involves measurement of the time that it takes to answer questions about the passages. The research will provide answers to two fundamental questions which have been the subject of substantial debate in recent years: (1) What role, if any, can general principles of memory play in explaining language processing, and (2) in what ways, if any, is reference to entities that are central (or topical) in a discourse processed differently from reference to entities that are not central.

DESCRIPTION (provided by applicant): This goal of this project is to increase understanding of the basic cognitive processes by which words are recognized during reading and to increase understanding of the neural basis of those processes. With respect to cognitive processes, the project focuses on how lexical representations (at the level of word form and word meaning) interact with the mechanisms of perception, attention and motor control that are critical during skilled reading, and further how representations at those lexical levels interact with the higher-level meanings created during discourse processing. With respect to the instantiation in the brain of cognitive processes, the project focuses on substantial divergences in conclusions about the nature of lexical processing that have emerged from studies that have used eye-tracking as a behavioral measure and those that have used event-related brain potentials (ERPs). Evidence from these methods has supported very different conclusions about the speed of word recognition and about the degree to which the processing of a word integrates a broad range of different types of information. The project addresses these issues in experiments that closely coordinate the use of behavioral measures (eye-tracking during reading) and neurophysiologic measures (ERPs). The project aims to fill gaps in current understanding about the following issues: (1) lexical access during reading, (2) the interaction of word-form representations and discourse representations, and (3) the interaction of word- meaning representations and discourse representations. Achieving these aims will contribute to theories of human language by allowing integration of current understanding about word-level and sentence-level processes, and deepening understanding of cognitive and neural architectures in a central domain of language processing. In addition the project will address important gaps in methodological knowledge about the sensitivity of behavioral and neurophysiologic measures, and about how to combine behavioral and neural evidence about language processing. The project will also contribute to the diagnosis and remediation of impairments in language and memory by deepening understanding of the constructs and methods that are at the forefront of efforts to determine how cognitive processing is affected by normal aging, dementia and aphasia. PUBLIC HEALTH RELEVANCE: The process of understanding language depends on memory to store information about partially-interpreted linguistic input, and long-term memory ability is greatly enhanced by the use of linguistic information to organize information that must be remembered. The results of the project will provide basic information about this interaction of language and memory, and in doing so will contribute to the development of better techniques for diagnosis and treatment (or remediation) of a range of problems that can occur in language abilities, including difficulty in learning to read, loss of language abilities due to neurological damage, and impairment of memory function due to aging or neurological damage.

DESCRIPTION (provided by applicant): This goal of this project is to increase understanding of the basic cognitive processes by which words are recognized during reading and to increase understanding of the neural basis of those processes. With respect to cognitive processes, the project focuses on how lexical representations (at the level of word form and word meaning) interact with the mechanisms of perception, attention and motor control that are critical during skilled reading, and further how representations at those lexical levels interact with the higher-level meanings created during discourse processing. With respect to the instantiation in the brain of cognitive processes, the project focuses on substantial divergences in conclusions about the nature of lexical processing that have emerged from studies that have used eye-tracking as a behavioral measure and those that have used event-related brain potentials (ERPs). Evidence from these methods has supported very different conclusions about the speed of word recognition and about the degree to which the processing of a word integrates a broad range of different types of information. The project addresses these issues in experiments that closely coordinate the use of behavioral measures (eye-tracking during reading) and neurophysiologic measures (ERPs). The project aims to fill gaps in current understanding about the following issues: (1) lexical access during reading, (2) the interaction of word-form representations and discourse representations, and (3) the interaction of word- meaning representations and discourse representations. Achieving these aims will contribute to theories of human language by allowing integration of current understanding about word-level and sentence-level processes, and deepening understanding of cognitive and neural architectures in a central domain of language processing. In addition the project will address important gaps in methodological knowledge about the sensitivity of behavioral and neurophysiologic measures, and about how to combine behavioral and neural evidence about language processing. The project will also contribute to the diagnosis and remediation of impairments in language and memory by deepening understanding of the constructs and methods that are at the forefront of efforts to determine how cognitive processing is affected by normal aging, dementia and aphasia. PUBLIC HEALTH RELEVANCE: The process of understanding language depends on memory to store information about partially-interpreted linguistic input, and long-term memory ability is greatly enhanced by the use of linguistic information to organize information that must be remembered. The results of the project will provide basic information about this interaction of language and memory, and in doing so will contribute to the development of better techniques for diagnosis and treatment (or remediation) of a range of problems that can occur in language abilities, including difficulty in learning to read, loss of language abilities due to neurological damage, and impairment of memory function due to aging or neurological damage.

DESCRIPTION (provided by applicant): The activity of reading raises fundamental theoretical and practical questions about healthy cognitive aging. It relies greatly on knowledge of patterns of language and of meaning at the level of words and topics of text. Further, this knowledge must be rapidly accessed so that it can be coordinated with processes of perception, attention, memory and motor control that sustain skilled reading at rates of four-to-five words a second. As such, reading depends both on crystallized semantic intelligence which grows or is maintained through healthy aging, and on components of fluid intelligence which decline with healthy aging. Successful reading is important to older adults because it facilitates completion of everyday tasks that are essential to independent living and because it entails the kind of active mental engagement that can preserve and deepen the cognitive reserve that may mitigate the negative consequences of age-related changes in the brain. The aims of the study are to determine how aging interacts with individual differences in cognitive abilities to affect word recognition durin reading and the relation between sentence comprehension and memory for text. These aims will be pursued: (1) by using individual-differences measures that are largely unexploited in the study of aging, (2) by evaluating alternative accounts of why characteristics of eye movements related to word recognition differ in older adults as compared to younger adults, and (3) by evaluating whether a cue-based long-term-working-memory framework can explain how aging affects both the process of sentence comprehension and subsequent memory for text. This project takes an innovative approach to aging and reading in the ways that it assesses individual differences and in how it characterizes and assesses both online reading strategies and the relationship between reading comprehension and memory. The results of the proposed research will advance theoretical understanding of the effects of aging on word recognition during reading and on the relationship between language comprehension and memory. The research will also facilitate future inclusion of precise measures of reading in both basic and applied studies of cognitive aging.