Fernanda Ferreira - US grants

9319272 Ferreira ABSTRACT The purpose of this project is to explore the architecture of the language processing system by examining how prosody is created and used. More specifically, the goals are to: (1) elaborate a model of prosodic production proposed recently by Ferreira (1993), which in turn will shed light on more global models of production; (2) use this model to develop and test hypothesis about how prosodic information might be used and integrated with syntactic and semantic information during auditory language comprehension; and (3) develop an online task for examining auditory language comprehension, an auditory analogue of the moving window task. In addition, by combining research in production and comprehension, this project will shed light on the important question of how the language production and comprehension systems are related. The proposed studies are divided into two sections, one on production and one on comprehension. The production experiments employ a paradigm in which participants view a picture or answer a question and generate a sentence in response to the stimulus. These studies designed to explore the prosody of spoken sentences employ a paradigm in which speakers read a sentence, memorize it, and produce it upon receipt of a cue. These experiments will examine the syntactic and phonological factors influencing the durational and pitch properties of spoken sentences. The comprehension experiments will use a technique Ferreira has recently developed called the Auditory Moving Window, which measures processing load across a sentence. (To allow comparison of sentence processing in the auditory and visual domains, the comprehension studies will also be conducted using eye movement monitoring and the visual moving window paradigms.) The listeners will hear sentences containing prosodic cues normally produced by speakers for those sentences, either with or without biasing lexical and contextual information. The goal of these studies is to examine how t hese sources of information interact during auditory language processing.

Ferreira
99-76584

Abstract

The correct interpretation of a sentence requires, among many other things, knowing which objects or actors named in the sentence played which roles in the event the sentence describes. In the sentence "The man bit the dog", for example, one needs to understand who did the biting in order to understand the sentence correctly. This interpretive process is referred to as binding thematic roles to concepts. The goal of this project is to develop and test a theory of how people do this in comprehending sentences communicated to them by others.

Pilot work has shown that the process is not nearly so simple and straightforward as previous research has assumed. Even unambiguous sentences present surprising difficulties of interpretation. For example, undergraduate college students hearing the sentence "The dog was bitten by the man" misidentify the biter about a quarter of the time, even though they almost always correctly interpret "The man bit the dog".

The project breaks down into three components:

(1) To develop a mechanism for assigning thematic roles through syntax, which can explain how phrases become vulnerable to misinterpretation.
(2) To provide an account of how that mechanism works with a parallel system that provides information about thematic roles via schematic knowledge.
(3) To provide evidence regarding the kinds of influences that can exaggerate or ameliorate thematic misinterpretation, including linguistic, schematic, and individual difference variables.

Formal linguists and psycholinguists start with the standard assumption that all human brains operate according to the same principles, and that all human language and human sentence processing must adhere to these principles, regardless of the surface characteristics of any given language. Yet the vast majority of psycholinguistic research (on both production and comprehension) that has been conducted has focused on a tiny number of the world's languages. With only a few exceptions, all of the languages have been typologically homogeneous and therefore syntactically similar. The proposed research seeks primarily to extend psycholinguistic research to the Odawa language, a Native American language closely related to Ojibwa and other Algonquian languages and spoken in southern Ontario, Canada. Odawa is typologically and syntactically very different from any language ever investigated by psycholinguists and thus offers a totally new perspective on the various current and competing theories of human sentence processing.

Odawa is often referred to as a nonconfigurational language due to its extremely free word order in independent clauses. Any transitive sentence consisting of two noun phrase arguments and a verb can occur grammatically in any of the six possible word orders. Yet this word order is not facilitated by a rich, conventional case system of the sort observed in languages that exhibit certain degrees of word order freedom and which have been studied by psycholinguists (e.g., German, Japanese). This degree of word order freedom is something that is not taken into consideration in any current model of sentence production or comprehension.

The Student Investigator has the unprecedented opportunity to combine his training in formal syntax, psycholinguistics, and the Odawa language to perform a large number of production and comprehension experiments in the town of Wikwemikong, Ontario, Canada, where approximately 50% of the 7,000 residents are still native speakers of the language. The production experiments will be conducted to assess baseline preferences for various structural alternatives, and to determine whether the preferences can be changed with various priming manipulations. The comprehension experiments will seek to establish temporary ambiguities and then observe the manner and time course of participants' recovery from such ambiguities. Critically, the syntax of Odawa allows key assumptions made by current theories of sentence processing to be critically examined.

Ferreira
0083570

The grant provides funds to allow the investigators to obtain significant instrumentation for cognitive / behavioral research. The requested-funds wo6/ld be used to purchase a free-viewing, mobile eye movement monitoring system, along with a computer workstation and a monitor for displaying some stimuli (in other cases, stimuli will be real, three-dimensional objects and scenes). The mobile eyetracker allows the viewer to examine real-world scenes or visual displays of scenes while making normal, natural head and body movements. Indeed, the system allows researchers to obtain precise information about where a person is looking as he or she moves through or manipulates objects in a natural environment. The instrumentation would be used for a variety of studies in cognitive and behavioral sciences. These include: (1) Research to examine how comprehenders quickly obtain interpretations for spoken sentences. No other existing methodology allows researchers to measure moment-by-moment processing for aurally presented sentences. (2) Research to study how real-world scenes are represented, and how representations of objects and scenes are generated dynamically over time in the context of meaningful actions. (3) Investigations of how humans are able to navigate novel and familiar environments, focusing particularly on eye movement patterns (e.g., what objects are used as guideposts and landmarks). (4) Studies of human-computer interaction, including the representation of objects and navigation through "virtual reality" environments. The free-viewing eyetracker would complement the Principal Investigators' existing laboratory facilities and greatly enhance the ability to train undergraduate and graduate students in sophisticated methodologies for studying complex behavior in intelligent systems.

This IGERT project examines the problem of sequential decision-making as a unifying framework for the study of several central topics in cognitive science: selective attention, navigation, language processing, and the coordination of action in multiple-agent groups. The overarching question our students are trained to investigate is the following: how is it possible for an agent to decide what actions to take to achieve long-term goals? We recognize that decision-making in complex environments is a sequential process, involving a series of episodes in which an agent, based on information available through its senses and stored in memory, selects the action appropriate for its goals. The problem is made difficult by perceptual uncertainty arising from sensory limitations and environmental complexity, by the challenge of sorting through the large space of actions available, and by inherent delays in feedback about the long-term consequences of actions. A wide variety of fundamental cognitive tasks can be cast as sequential decision-making problems. Understanding how such problems may be solved will be a critical component of a general theory of intelligent behavior in organisms, and will be essential for the design of truly intelligent machines. To study these problems, we adopt a comparative approach, combining insights from a range of model systems, including humans, non-human animals, robots, and intelligent software agents. This multidisciplinary framework will enable students to integrate ideas and methods from different fields that have been concerned with the study of sequential decision-making (psychology, behavioral biology, linguistics, and computer science), but that have so far remained largely separate. The training program is designed to create a new generation of scientists trained in this innovative, multidisciplinary approach. Graduate training will be focused on fundamental disciplinary education, a common set of courses focused on the sequential decision-making framework, and a strong emphasis on mentored, interdisciplinary research activities that span each student's entire graduate program.

IGERT is an NSF-wide program intended to meet the challenges of educating Ph.D. scientists and engineers with the multidisciplinary backgrounds and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing new, innovative models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries. In the fourth year of the program, awards are being made to twenty-two institutions for programs that collectively span all areas of science and engineering supported by NSF. The intellectual foci of this specific award reside in the Directorates for Social, Behavioral, and Economic Sciences; Computer and Information Science and Engineering; Engineering; Biological Sciences; and Education and Human Resources.

In spoken dialog systems, interpreting user speech input is still a significant challenge due to limited speech recognition and language understanding performance. This problem is further amplified if a user has an accent or is speaking in a noisy environment. However, previous research has shown that, in multimodal systems, fusing two or more information sources can be an effective means of reducing recognition uncertainties, for example through mutual disambiguation. Inspired by earlier work on multimodal systems, in this project the PI will investigate the role of eye gaze in human machine conversation, in particular in salience modeling for robust spoken language understanding. Cognitive studies have shown that human eye gaze is one of the reliable indicators of what a person is "thinking about." Specifically, eye gaze is tightly linked to human language processing. Previous psycholinguistic work has shown that almost immediately after hearing a word, the eyes move to the corresponding real-world referent. And right before speaking a word, the eyes also move to the mentioned object. Not only is eye gaze highly reliable, it is also an implicit, subconscious reflex of speech. The user does not need to make a conscious decision; the eye automatically moves towards the relevant object, without the user even being aware. Motivated by these psycholinguistic findings, the PI's hypothesis is that during human machine conversation user eye gaze information coupled with conversation context can signal a part of the physical world (related to the domain and the graphical interface) that is most salient at each point of communication, thus it can potentially be used to tailor the interpretation of speech input. Based on this hypothesis, the PI will seek to improve spoken language understanding in conversational interfaces through a new salience-based framework with two objectives: (1) To better understand the role of eye gaze in human language production and its implications in salience modeling for automated input interpretation; and (2) To develop algorithms and systems that apply computational gaze based salience modeling to robust spoken language understanding. These objectives will be pursued in the following four directions: (a) Investigation of the utility of human eye gaze and its implications for salience modeling during human machine conversation through psycholinguistic studies; (b) Development of computational salience models that integrate eye gaze with conversation context to automatically identify a salient part of the physical world at each point of communication; (c) Development of approaches that apply the new salience models to constrain the hypothesis space for robust spoken language understanding; and (d) Evaluation of the generality of the new approaches in two different applications: an interior design/training application based on a 3D rendered interface, and an information seeking application using a 2D map-based interface.

Broader Impacts: The technologies to be developed in this interdisciplinary project can be applied to many applications such as virtual training systems where users can see the interface and talk to the computer system at the same time. The technologies will benefit a variety of diverse users, and particularly individuals who are unable to interact with graphical interfaces with their hands (e.g., motion disabled users). Since one major application area of the work is e-training and e-learning, the education and outreach impact of the proposed research is potentially profound; the PI will make specific efforts to transfer the research results into classrooms. The project will also provide a unique opportunity for students in Computer Science, Psychology, and Cognitive Science to work together, and thus will synergize multidisciplinary research activities at Michigan State University.

The cognitively based theories that influence current psycholinguistic research are based largely on non-neural data and theoretical constructs. The foundations of these theories are formal linguistic models of grammar as well as cognitive models of memory, attention, and learning. However, given that language processing must take place in a physical structure, it is critical to develop theories that are biologically plausible and compatible with other theories in the cognitive neurosciences. The 26th Annual CUNY Human Sentence Processing Conference, to be held in March 2013 at the University of South Carolina, will include a Special Session to address this question: Can the basic architecture of language developed in the 1950s and 1960s that was based primarily on linguistic evidence, or in the 1980s and 1990s based on statistical constraint-based models, survive in the era of brain imaging, brain stimulation, and sophisticated cognitive neuropsychology? If not, what new architectures for the language system are compatible with what has been learned from the entire range of relevant evidence--including linguistic, behavioral, and biological data?

The Special Session will bring together six prominent researchers to consider this fundamental issue. This particular group was chosen because a range of major cognitive neuroscience methodologies besides traditional neuropsychology is represented, and because the speakers take diverse theoretical perspectives on linguistic architectures and processing.

The Special Session will make contact with the widest possible spectrum of conference attendees, but particularly with younger scientists with interests in neurobiological approaches to cognition and language. The Special Session will also help to ensure that the resource-intensive research undertaken by neuroscientists studying language will have as broad an impact as possible and will begin to seriously inform fundamental theorizing in all areas of psycholinguistics.

DESCRIPTION (provided by applicant): Everyday speech is often produced imperfectly, requiring listeners to navigate a stream of input interrupted by disfluencies, and these can affect how spoken language is processed and ultimately interpreted. Disfluencies may impede communication in a wide range of settings, from customer service to medical provision to public safety. Some disfluencies are unproductive, while others, including self-corrections, may play a crucial role in managing and maintaining the communication of information. However, in general, little is known about how self-corrections from healthy adults and individuals with a speech disorder such as stuttering are processed. Two contrasting theoretical approaches to this question are the ambiguity resolution and noisy channel models of disfluency processing. The main aim of this project is to investigate disfluency processing with an innovative, mixed-methods approach using utterances spoken by healthy individuals and people who stutter. The experiments use both naturally generated disfluent speech and constructed examples, and the constructed examples will be elicited from both people who do and do not stutter. The experimental paradigms to be used are the most sophisticated available for examining the online processing of speech: recording eye movements to depicted referents mentioned in spoken utterances (the Visual World Paradigm), and recording Event-Related Potentials in response to specific properties of spoken utterances containing self-corrections. Participants will be healthy college students from the University of South Carolina and the University of South Florida undergraduate communities. The Specific Aims are: (1) To use neural methods and eyetracking to systematically investigate the factors that allow listeners to process self-correction disfluencies in real time; and (2) to further discriminate between the ambiguity resolution and noisy channel models by using speech from speakers with different production systems: people who do and do not stutter. An additional goal is to provide preliminary information about the possibility of assessing therapeutically modified speech using a methodology based on measures of online spoken language processing. The project is highly innovative because of (a) the focus on self-correction disfluencies and the use of online techniques (eyetracking and electrophysiology) to examine how they are processed as they are encountered in real time; (b) the testing of theories which assume that the same tools that are used by comprehenders to process regular utterances are also used to handle self-corrections, avoiding the need for special-purpose mechanisms; and (c) the examination of how speech produced by people who stutter is processed in real-time by healthy adults, which may lead to an implicit, performance-based instrument for assessing the success of therapeutic interventions to treat stuttering.

Everyone is familiar with the experience of being disfluent. Despite their best efforts, on average speakers will produce a filler such as "uh" or an "um" every twenty words, and they may also make a speech error which will need to be repaired. Previous research has established some of the causes of disfluency and has revealed that disfluencies of different types characterize different types of speakers; for example, individuals with ADHD are more likely to produce speech errors than to use fillers, and those without ADHD show the opposite pattern. Far less is known, however, about how listeners are able to understand speakers despite the presence of this noise in the linguistic signal. Early proposals tested the hypothesis that listeners must somehow ignore disfluencies, but more recent studies show that disfluencies are only partially suppressed, indicating that disfluencies affect how listeners interpret the sentence they hear and even how they evaluate the speaker. In addition, these newer experiments show that when listeners hear a word spoken in error, they use the error to predict what the speaker is likely to say instead. This prediction mechanism is helpful for two reasons: first, because it allows the listener to get a head start on processing the speaker's intended meaning; and second, because it helps the listener come up with a more sensible interpretation of the utterance should the speaker fail to detect and correct his or her error. Understanding how these prediction mechanisms operate is especially relevant for our understanding of language and aging; speakers are known to become more disfluent as they age, making their speech harder to understand. This is a pressing concern given the aging population of the United States. This work will also help enhance speech recognition devices that must be robust to disfluency if they are to operate on natural, spontaneous speech. Devices that respond to voice commands are now in millions of Americans' homes and pockets, and as they become more common, users will increasingly come to expect them to work smoothly and reliably.

The experiments that will be conducted for this project use two complementary methods for assessing people's comprehension of speech on a millisecond-by-millisecond basis: recording of brain electrophysiological activity (EEG), and recording of eye movements to visual displays presented during listening tasks. The experiments are designed to answer three core questions about prediction during processing of disfluencies: (1) When do listeners begin to predict? (2) What precisely is the content of the prediction (a specific word, a general category)? (3) What is the fate of an incorrect prediction? That is, given that listeners' expectations will not always match the speaker's output, how do listeners reconcile their prediction with any discrepant content? This project will involve students who will be trained in experimental psycholinguistics, statistics, and computational methods, allowing them to gain experience in designing and interpreting data, as well as in preparing scientific reports for presentation and publication.

Project Summary Efficient cognitive processing relies on the brain?s ability to engage in prediction and to use forward modeling to anticipate cognitive events, including during language processing. The central goal of this proposal is to test two competing hypotheses concerning how age influences prediction during auditory and visual language processing. Current evidence is contradictory and sparse, reflecting the need for systematic investigation. The project has three Specific Aims: Aim1: Determine whether older adults predict words in manipulated sentence contexts less or more than younger adults do by examining prediction during reading, using both electrophysiology (EEG) and eyetracking methods. Aim2: Determine whether, during spoken language processing, older adults predict words in manipulated sentence contexts less or more than younger adults do, using EEG and Visual World eyetracking methods. Spoken language processing merits targeted investigation because evidence suggests older adults have specific problems with auditory input. Moreover, in the young adult literature on prediction in language processing, relatively few studies have focused on spoken language, so little is known about whether prediction differs in the two modalities. Aim3: Determine whether older adults predict upcoming words in connected passages less or more than younger adults do, using fixation-related fMRI and EEG methods in reading, with prediction assessed by continuous measures of lexical surprisal and entropy. Surprisal and entropy measures permit the investigation of more naturally varying levels of predictability, more natural distributions of predictable and less predictable information, and allow the investigation of how natural texts (i.e., stimuli not specifically created for an experiment) are comprehended. Innovations: The project is innovative in (1) the use of converging eyetracking, EEG, and fMRI methods to systematically evaluate the extent of prediction during older adults' language comprehension, emphasizing replication across techniques and modalities; (2) the use of continuously varying surprisal/entropy in connected text to index age differences in prediction; (3) the use of a novel technique developed by PI Henderson, Fixation-Related fMRI, to relate neural activation to word-by-word surprisal and entropy during natural reading. Significance: The experiments will yield high temporal resolution information about prediction in older adults during online reading and spoken comprehension, together with detailed information about the neural bases of prediction operations. The findings have important implications for theories of normal cognitive aging. Translational significance: A psychometrically valid assessment of everyday language skills will be used to evaluate the relationship between prediction skills and a measure that has been shown to predict impairments associated with Alzheimer?s disease. Overall, prediction in language processing is potentially a model system for enhancing our scientific understanding of how cognitive and neural decline associated with aging trades off against greater knowledge and experience.

To tell a story, give directions, or describe the layout of a house, speakers must generate multiple utterances in sequence. Because most psycholinguistic research on speaking is based on paradigms designed to elicit single utterances, little is known about multi-utterance language production in children and adults. Linked to this empirical focus on single utterances is the widespread use of a method in which subjects describe simple visual images containing just a handful of objects shown in straightforward spatial arrangements. In contrast, real world scenes contain multiple objects in various relationships that can be described in numerous ways, and so the attentional and language systems face a challenging set of decisions concerning where to begin the description, how to cluster objects and capture their relations, and what information to include or omit. This project uses complex, real-world scenes as a tool to examine the linearization of complex thoughts into sequenced utterances, focusing on adults at this investigate stage in order to establish developmental benchmarks. Image and semantic characteristics of complex scenes will be precisely quantified and used to generate predictions about the allocation of attention as a scene is viewed and described. The project examines the conditions under which scene image features exogenously draw the eyes to specific visual areas which the linguistic system then describes, and under what conditions the cognitive system guides the eyes to meaningful regions of the scene, allowing the language system to prepare a description even before the relevant object or region has been fixated. On this latter view, the language and cognitive systems use scene meaning interactively to formulate a linearization plan for coordinating the production of multiple utterances. To address these theoretical issues, the project focuses on three Specific Aims: (1) To use computational tools from the field of visual cognition to measure image and meaning properties of complex scenes, which will permit the precise quantification of features controlling attention during speaking tasks as well as the selection and sequencing of linguistic content. (2) To determine the extent to which viewers predict the presence of objects and their locations and use those predictions to get a head-start on linguistic encoding even before an object is attended. (3) To extend our approach to the production of utterances describing events by applying the same methods for quantifying scene image and meaning properties that have been developed for nonevent scenes to scenes depicting events with and without animate agents. The project tests an innovative theory of multi-utterance language production which assumes that speakers formulate a linearization plan to guide the allocation of attention and linguistic decisions concerning inclusion and ordering of information. This approach will lead to a deeper understanding of language production, which will lead to identification of testable, rigorous hypotheses concerning the emergence of these processes across development.