Michael K. Tanenhaus - US grants

Recognizing the words in a text is an important part of skilled reading and a source of difficulty for many children who are problem readers and for many adults who have acquired language disorders as a result of stroke or other brain injury. An important question for research on word recognition is when are words decoded wholistically as complete units and when are they decoded compositionally by analysis into sub-lexical units. The proposed research addresses this question for words which are morphologically complex, that is words which contain more than one morpheme. The general hypothesis being explored is that high frequency words will be analyzed wholistically while lower frequency words will be analyzed compositionally. In addition, it is expected that whether or not a lower frequency word will be processed compositionally will depend on the relationship between the frequency of the word and the frequency of its parts and the degree to which the meaning of the word is predictable from the meaning of its component morphemes. It is also expected that there will be developmental differences in how these words are processed with younger and less skilled readers relying more heavily on a compositional strategy than older and more skilled readers. These hypotheses will be explored in a series of studies using lexical decision and naming tasks with skilled adult readers and children of various ages and reading abilities. The long-term goals of this research are to understand how word recognition is accomplished in skilled reading, how word recognition skills develop, and why certain children have difficulties with word recognition. Understanding how morphologically complex words are processed is an important step in reaching these goals.

The answer to fundamental questions about the organization of the language comprehension system depends upon obtaining information about how different types of knowledge are represented, accessed, and integrated during comprehension. The speed with which coprehension takes place neccessitates obtaining detailed information about the time course of comprehension. The proposed research explores the use of the cortical evoked potential as a technique for studying comprehension. Two sets of experiments are proposed. The first experiments examine the parsing process. Recent research using evoked potentials has identified a component of the waveform, N400, that is sensitive to whether or not a word is congruous with its context. Materials are developed in which alternative parsing models make different predictions about when and where in sentences local incongruities should develop. N400 is then used to test these predictions. The second series of experiments examine how context influences the recognition of a word by determining when the waveform of word and nonword stimuli diverge when words are presented in different types of linguistic context. These experiments will help us to better understand the organization of the language comprehension system. They will also provide a detailed evaluation of the value of the evoked potential as a methodology for studying comprehension processes. Finally, the use of well-defined linguistic stimuli in these experiments will help us to better understand how certain components of the evoked potential are related to cognitive processing.

This project will continue long-term research in psycholinguistics to advance our knowledge of the cognitive processes involved in understanding the language of others. The PIs' earlier work has shed significant light on how we understand anaphoric or pronominal references and their relationship to their antecedents in the context of a discourse, and how we decide which of various possible readings of a word or phrase is the right one in a given context. The latter work has led to the focal topic of the current research. Results of their experiments and those of others, coupled with recent developments in linguistic theory, have suggested that the so-called "thematic" structure of verbs plays a key role in interpreting sentences containing those verbs. The investigators have developed a model of the language-understanding process, centered on thematic structures, that makes strong empirical predictions. The current research is a series of experiments designed to test those predictions.

Readers and listeners make provisional or partial commitments at multiple levels of representation as linguistic input unfolds over time. The continuousness of comprehension raises methodological and theoretical challenges. The methodological challenge is to monitor comprehension processes as they are occurring. The theoretical challenge is to develop and evaluate explicit mechanistic models that support real-time comprehension. Such models are essential for understanding normal comprehension and its development, developmental disabilities, and comprehension impairments that arise from brain injury. The goal of this project is to develop and test explicit models of how readers and listeners coordinate linguistic and non-linguistic contextual information in constructing interpretations during real-time language comprehension. The proposed research examines the mechanisms underlying real-time language comprehension within a constraint-based framework emphasizing rich lexical representations and the incremental updating of a discourse model. Two lines of research are proposed. The first extends constraint-based models of ambiguity resolution to evaluate whether they can account for processing differences between different classes of verbs without direct appeal to syntactic complexity. These studies use corpus analyses, ratings, fragment completion, computational modeling, and reading time experiments. The second line of research examines how listeners establish, update, and refer to referential domains as an utterance unfolds over time, and how referential domains affect syntactic ambiguity. Reference must be contextualized to a relevant domain of interpretation. Although this domain ultimately includes information introduced linguistically into the discourse, salient objects in the environment, and shared presuppositions between participants in a conversation, little is known about how and when these different sources of constraint are used. The proposed work uses a head-mounted eye-tracking paradigm in which participants follow spoken instructions to move real objects in a workspace or pictures displayed on a monitor. By manipulating the objects in the display and the instructions, it is possible to determine the time course with which semantic and pragmatic constraints are used in circumscribing referential domains.

ABSTRACT The annual CUNY sentence processing conference will take place at the University of Rochester this year. This conference assembles 100-200 researchers active in the study of language processing, including linguists, computational linguists, psychologists and philosophers. The conference typically has had invited papers and a fairly open poster session, primarily for graduate students. This year, the organizers are emphasizing increased participation, including a special university-sponsored fund for about 30 graduate student fellowships, several sessions open to competitive submissions and invited posters from senior investigators. There will be three didactic symposia, each devoted to presenting a current area of research which should have a large impact on future sentence processing research. (1) A second wave of connectionist modelling offers new ways of thinking about how to model processes; (2) recent research has coalesced around three different kinds of methods for studying on-line sentence processing, word reading time, on-line probe responses and evoked response potentials on the skull; (3) fresh ideas about lexical concepts and how they are accessed at different levels may lead to models which better integrate the processing of conceptual and structural information. In each case, the symposium will have a main presenter whose charge is to lay out the issues, theories and current results in a way which sentence processing specialists can understand. Commentators have been chosen to explore specific directions and to make specific proposals about the implications for processing research.

This project examines the real-time processing of sentences with focus constructions within the framework of a representational theory of the semantics of focus. We will concentrate primarily on sentences and discourses exhibiting contrastive focus, with special emphasis on the theoretical notion of a contrast set, the group of alternative entities, properties, etc., that something in the assertion in the sentence implicitly stands in contrast to. Existing literature clearly establishes that readers and listeners devote extra attention to focused words and phrases. However, to our knowledge, there has been little psycholinguistic work to date conducted within the context of an explicit representational theory of focus. Consequently, fundamental questions about the informational contribution of contrastive focus, and its consequences for on-line language processing, have not yet been explored. The research that we are planning investigates the processing of sentences and discourses exhibiting contrastive focus, as understood within the context of current semantic theory. The most basic question is whether contrast sets are indeed computed during the course of understanding an utterance marked with focus against the background of a discourse context, and, if so, when are they computed, and how are they computed (that is, what factors govern their determination). We will also examine how they are used in managing attention (and salience of entities) in discourse processing and interpreting referential expressions. Most of these questions will be examined using a new experimental paradigm in which subjects follow spoken instructions to manipulate objects in a workspace while their eye-movements are monitored using a video-based camera system. Preliminary work using the visual world paradigm has demonstrated that listeners seek to establish reference incrementally, making eye-movements to relevant objects that are closely time-locked to referential expre ssion in the instructions. Subjects typically make an eye-movement to an object prior to reaching for it and the programming of the eye-movement to the target object occurs, on average, within several hundred milliseconds of the word in the instruction that makes that object unique with respect to the relevant visual alternatives.

DESCRIPTION (provided by applicant): This proposal requests continued support for an interdisciplinary predoctoral and postdoctoral training program in the Language Sciences. The training program, coordinated by the Center for Language Sciences, includes 14 faculty from the Departments of Brain and Cognitive Sciences (6), Computer Science (3), Linguistics (4), and Social and Clinical Psychology (1). These core faculty provide an unusually rich and comprehensive coverage of natural language processing and acquisition. Faculty expertise spans formal linguistic, behavioral, computational, and cognitive neuroscience approaches, providing trainees with diverse methodological tools and an emphasis on interdisciplinary approaches to the study of natural language. Major program research themes include real-time language processing and the acquisition of spoken and signed languages. We have attracted outstanding students to study these topics and have placed them in faculty positions at top universities. In the next grant period we will increase research training on an addition- al theme: language processing and acquisition in populations with altered communicative abilities, including deaf individuals, individuals with autism, and patients with neurological disease. Such training will provide our students with a broadened perspective on natural language and will permit them to apply cutting-edge techniques to the study of language when linguistic input or biological underpinnings are altered. Support is requested for five predoctoral and three postdoctoral trainees who will be trained in the interdisciplinary language sciences. Predoctoral trainees will have Brain and Cognitive Sciences or Computer Science as their home department, with training in these fields and in Linguistics; postdoctoral trainees will have their home in any of the three departments. All trainees will take a structured set of core courses, proseminars in the language sciences and in translational research, and will acquire expertise in at least two methodological approaches. Students will also be closely mentored in research on the mechanisms underlying language processing and acquisition, and will have new opportunities to discover how these findings can be used to study, and potentially improve, communicative disorders. RELEVANCE: Language is a central feature of human intelligence and is maintained under a wide variety of circumstances, including even profound deafness (when many individuals will develop signed rather than spoken languages). Our program trains students to conduct research on spoken and sign language processing, and to understand - and ultimately develop adaptations or treatments for - communication disorders.

Many aspects of language comprehension are closely time-locked to linguistic input. Thus psycholinguists have increasingly relied upon on-line experimental methods that provide fine-grained temporal information about language processing, e.g., monitoring eye movements during reading. However, most standard methods do not allow for continuous monitoring and cannot be easily adapted to natural situations. We are using the sensori-motor laboratory to monitor eye movements as subjects follow spoken instructions to manipulate objects (e.g., ``Put the apple that's on the towel in the box.''). With well-defined tasks, eye movements can illuminate the rapid mental processes that underlie spoken language comprehension. This approach can help explore topics ranging from recognition of spoken words to conversational interactions during cooperative problem solving. Preliminary experiments showed that participants process instructions incrementally, making saccadic eye movements to objects immediately after hearing relevant words in the instructions. When asked to touch one of four blocks differing in marking, color, or shape, with instructions such as ``Touch the starred yellow square,'' subjects made an eye movement to the target block an average of 250 ms after the end of the word that uniquely specified the target with respect to the visual alternatives (e.g., after ``starred'' if only one block was starred; after ``square'' if there were two starred yellow blocks). With more complex instructions, subjects made informative sequences of eye movements to objects relevant to establishing reference (e.g., ``Put the five of hearts that is below the eight of clubs above the three of diamonds.'').

Understanding the mechanisms by which people recognize spoken words in continuous speech is of central importance for theories of how language is processed, how it develops, and how it is affected by brain injury. An understanding of how people recognize words in continuous speech also provides valuable informative for researchers developing speech recognition technology and human-computer interaction systems. It is well-established that during spoken word recognition listeners evaluate the unfolding input by activating a set of potential lexical candidates which compete for recognition. However, numerous questions remain about how the set of possibilities is established and how it is evaluated during real-time processing. For example, little is known about whether or not people are able to use fine-grained acoustic differences during initial word recognition. Thus, it is not clear whether as the word `carpet` is heard in continuous speech, the word recognition system considers all words that begin with similar sequences of sounds, e.g., (car, card, etc.) or whether subtle differences in the length of vowels in one-syllable and polysyllabic words are used to restrict the set of alternatives. Questions like these have important implications for how we understand and model the word recognition system. However, our ability to answer these questions has been limited because few of the experimental methods sensitive spoken word recognition can be used with continuous speech in natural tasks. This is an important limitation because natural speech often occurs in noisy conditions, there is considerable speaker variability, and linguistic units, such as the beginning and end of a word are not clearly marked in continuous speech. The proposed research explores how candidate words are retrieved from memory and evaluated during continuous speech using: (a) experimental studies in English with digitized natural speech and synthesized speech; (b) computational modeling; and (c) experimental and computational explorations with artificial languages. The experiments measure eye-movements to objects in a circumscribed visual world, extending the methodology pioneered by the PI and his collaborators. Participants will follow spoken instructions to pick up and move (with a mouse) line drawings of concrete objects on a computer monitor (e.g., `Pick up the candy. Now put it above the circle`.). Preliminary studies have established that: (a) the pattern and timing of eye-movements are remarkably sensitive to the uptake of information, allowing for a detailed mapping of the nature of the candidate set and how it changes over time during continuous speech; and (b) there is a simple quantitative mapping from hypothesized underlying speech recognition processes to the probability of making an eye-movement to a target object, allowing for precise testing of different theories of word recognition. Moreover, the basic task, either with pictures or real objects, can be naturally extended for use with infants, young children, and neurologically-impaired populations. The project should result in both methodological advances and in a body of empirical data important for scientists studying normal and impaired language processing, as well as for scientists developing speech recognition systems.

With National Science Foundation support, Drs. Jeffrey T. Runner and Michael K. Tanenhaus will conduct three years of linguistic research examining how participants interpret two important classes of referring words: pronouns and reflexives. The project will use a lightweight head-mounted eye tracker to monitor eye movements as participants identify a picture in a scene while listening to descriptions of the depicted activities (e.g., "John told Bill about the picture of himself."). Which objects participants look at, and when they look at them, provides evidence about which referents they are considering as they interpret the referring words. The goals of this research are to (a) investigate the interpretation of these referring words with on-line tasks rather than explicit judgments; (b) evaluate the extent to which grammatical factors define the initial set of referents for pronouns and reflexives; (c) examine the interaction of grammatical factors with pragmatic and discourse factors; and (d) explore the extent to which the potential set of referents for pronouns and reflexives is mutually exclusive.

The results of this research will be important for understanding how people assign reference, which is a central goal of current research in theoretical and applied linguistics. Reference resolution is one of the most central and challenging problems in developing efficient language understanding systems, including systems that are being developed for bio-medical applications. Difficulties in reference resolution are also associated with some types of language difficulties that arise due to brain damage. Information about the timing of eye movements and speech will also inform scientists who are developing computer-based language understanding systems that use eye movements to help resolve ambiguous words and referring expressions.