John Trueswell - US grants

This project will examine how children develop the ability to rapidly interpret speech as it is being heard. To accomplish this research objective, a free-head eyetracking system will be used to monitor the eye movements of children (ages 4 years and up) as they respond to spoken instructions. By examining which objects a child considers as he or she is listening to speech, certain inferences can be made about the on-going processes that underlie language comprehension. Of particular interest is how the child interprets sentences that contain uncertainties or ambiguities regarding the grammatical relationships between words and phrases. Prior research has found that adults rapidly commit to a single interpretation of an ambiguous phrase by coordinating several key sources of information, including detailed knowledge of individual words, information from the prosody ("melody") of speech, and information from the situation or context. The current research examines how a child develops this rapid and relatively interactive processing system. By testing the child's ability to take advantage of these potentially useful sources of information, we can gain a better understanding of how language is represented, organized, and processed by the normally developing child. Answering questions about children's language processing abilities has long eluded researchers, primarily because of technical limitations on tests suitable to use with children. The free-head eyetracking technology provides a new window into on-going language interpretation, supplying insights into normal development that have potential applications to the treatment of language-developmental delays and pathologies.

This grant will fund a special conference session on 'world-situated language use in natural dialog', held in conjunction with the 14th Annual CUNY Conference on Human Sentence Processing. The conference, held March 15-17, 2001, at the University of Pennsylvania in Philadelphia, is the most prominent U.S. conference for the interdisciplinary study of human language understanding. On an annual basis, it brings together roughly 250 linguists, psycholinguists and computational linguists interested in detailed processing accounts of language comprehension and production.

The special session, entitled "Approaches to Studying World-Situated Language Use: Bridging the Language-as-Product and Language-as-Action Traditions," is designed as a step toward linking conversational/discourse research with the formal linguistic and mechanistic approaches typically found at the CUNY conference. Five prominent researchers working in this bridging area have been invited to give talks and participate in a panel discussion. In addition, peer-reviewed submitted talks and posters on this topic will be presented in accompanying sessions. It is hoped that by holding this symposium at the CUNY Conference, timely cross-disciplinary discussions will occur so to inspire a new generation of psycholinguistic and computational research on questions such as how natural utterances with disfluencies are processed, how information from context, gesture and linguistic input are combined in real-time processing, how interlocutors coordinate attention, and how these coordination processes impact real-time language processing commitments.

This Integrative Graduate Education and Research Training (IGERT) award supports a multidisciplinary graduate training program at the University of Pennsylvania designed to integrate the computational, cognitive and neuroscientific study of communication and communication systems, be they characterized as human-linguistic, animal or machine. The primary purpose is to create a new breed of communication scientists capable of integrating theoretical issues, methods, and formalisms that are currently distributed across graduate programs as diverse as anthropology, biology, computer science and engineering, linguistics, neuroscience, philosophy, and psychology. The intellectual merit consists of the two interrelated research themes that will unite and guide graduate training. The first theme emphasizes communication as a dynamical process, one that unfolds along multiple time scales varying from milliseconds (as in planning and understanding speech) to centuries (as in evolving dialects, languages, and systems of animal communication). The second theme emphasizes communication as a context-sensitive process, where contexts range from the physical setting and communicative history of a specific conversation, to the linguistic, social and technological assumptions of social groups. Trainees will be co-advised by a multidisciplinary faculty team and will commit to a five-year graduate training program, consisting of: (1) core disciplinary training in one of the current graduate programs above; (2) one-year cross-disciplinary training in a chosen second discipline, including completion of a publishable research project; (3) participation in a weekly interdisciplinary research meeting throughout the 5-year program; and (4) completion of an advanced course in the mathematical foundations of communication specifically designed for this program. Broader impacts of this program include applications in industry, technology, and clinical settings. IGERT is an NSF-wide program intended to meet the challenges of educating U.S. Ph.D. scientists and engineers with the interdisciplinary background, deep knowledge in a chosen discipline, and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing innovative new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries.

People talk about events in their visual worlds in a way that seems both natural and effortless. However, these events must be represented in spoken language as well as the language of thought, so to speak, and the processes by which people translate between these two languages remain a matter of debate despite decades of linguistic and cognitive research. This research question is complicated by the fact that languages differ considerably in the way they encode even the simplest and most familiar events. Such cross-linguistic variations are common throughout the world's languages, and they raise deep questions about the relationship between language and thought. In the context of event cognition, the question is whether there are universal components of event cognition, and whether the linguistic encoding of event information has an effect on the operation of those components.

With support from the National Science Foundation, Dr. Papafragou and Dr. Trueswell will conduct a series of experimental studies with adults speaking different languages to study how dynamically unfolding events are mapped onto linguistic representations, and whether these representations affect the conceptual organization of the visual world. Their research offers an innovative and multi-pronged approach to one of the most fundamental topics in the study of the mind, namely how human perception and cognition make contact with the language faculty. Because of its thoroughly interdisciplinary nature, this project provides unique training for graduate and undergraduate students in the cognitive sciences, including eye-tracking, cross-linguistic, and international work.

DESCRIPTION (provided by applicant): This project aims to discover how languages are learned and understood at the levels of both word and sentence. The first problem we address is understanding how learners find out what words such as "dog" and "cat" signify. The second problem is learning how these words combine semantically in sentences such as "The dog bites the cat" versus "The cat bites the dog." Though the words in this case are all the same ones, in English their ordering (more precisely, the structure that binds them) determines their semantic roles (do-er or done-to) with respect to the "biting" act. Some languages rarely use this ordering method and if they do their orders may be different, so learners of English have to acquire these properties by analyzing the speech that they hear from adults to derive the English facts. In most cases children don't get explicit instruction about most of the word meanings or about the syntax, but they learn even so. The learning process applies not only to children but also to learning of second languages by older individuals, including adults. Much of the work proposed herein uses a relatively new experimental technique, developed in our laboratories under earlier funding of this grant, in which children's eye gaze is tracked as they hear spoken descriptions of the surrounding visual world. Specifically, children hear instructions that require them to make an implicit choice about the intended structural organization of ambiguous utterances such as "I saw the man with a telescope." By manipulating potentially informative cues to the intended structure (e.g., verb information, prosodic (tune) information and situational/discourse cues), children's eye gaze and other behaviors can reveal their sensitivity to and representation of these information sources. In the upcoming funding period, we propose: (a) To expand and test our developmental account of how children learn to recover the grammatical properties of a sentence as it is heard, by examining eye gaze responses to ambiguous sentences at different ages;(b) To explore how multiple linguistic and non-linguistic cues regarding speaker's intentions are used by the child to uncover word and sentence meaning;(c) To examine what is tracked by the child regarding the meaning of verbs and other relational lexical items as they hear them;(d) To examine how sentences understanding procedures are learned and used in languages that are quite different from English in the clues to meaning that they offer (specifically, Korean, Tagalog, and Kannada, and perhaps two others). The potential applications of these findings to education are significant, as vocabulary and sentence understanding skills are fundamental to successful functioning in the technological culture of the 21st century, and many children are in need of enhancing and remedial intervention. In addition, as the United States citizenry becomes progressively more multilingual, and is increasingly drawn into global interactions, the ability to acquire second, and even third and fourth, languages becomes an ever more precious social and economic commodity PUBLIC HEALTH RELEVANCE This project is designed to further the understanding of how young children learn what the words in their language mean, and how these words are combined to make meaningful sentences. The ability to understand spoken and written language rapidly and close to errorlessly is a basic requirement for economic and social well-being in 21st Century American life. The findings are expected to be relevant to second language learning as well;multilingualism is an increasingly precious commodity for Americans as they interact more and more with speakers of different languages both with the country and with cultures around the world.

Project Summary The goal of this project is to understand how children (and older second-language learners) discover the meanings of words and their semantic roles in sentences. Even for simple words like dog it is not easy to glean what they mean from observing (as one usually will) that there is a dog in sight. For after all, when a dog is in sight, so is his fur, and his ear, and the joyous wagging of his tail. To which of these factors or properties does the pointing finger refer when an adult says Look at that doggie! Things get even harder when the child has to learn the meanings of words like idea or think because in these cases there's nothing so obvious to point to out there in the world. Yet children of three and four years of age understand and utter such apparently abstract words. Our projects take off from the observation that not all words occur in the same places in sentences, for instance one can say I think (or see) that you're cute but not I jump that you're cute. Strikingly, children as young as two- and three-years of age are sensitive to these positional privileges, which in turn give clues to word meaning. Because it is hard to get information about children's word meanings or their learning by asking them for definitions or the like, we use implicit methods such as tracking children's eye gaze direction and responsiveness to queries that place words into different visuo-social environments and into different syntactic structures to find out about their evolving word knowledge. These issues are of much more than academic interest. No tested property of child cognition or behavior is a better predictor of school and work-place success than vocabulary growth in the first few years of life. Vocabulary scores diverge for children of higher or lower socio-economic status as early as the second birthday, and these differences increase throughout the school years, influencing all the child's subsequent learning. So our work extends to discovering ways that actual home and early school environments can maximize the supportive environment for vocabulary and syntax acquisition. Our past work gives strong indications of what these favorable learning environments are, and the present proposal inquires more deeply into these factors.

DESCRIPTION (provided by applicant): Many highly proficient bilinguals engage in spontaneous code-switching, fluidly alternating between their two languages when conversing with each other. Experimental methods that have investigated language switching in bilinguals consistently reveal costs (i.e. longer naming times) to switching. The experimental evidence stands at odds with the seamless integration and frequency of use of both languages in spontaneous code-switching. This proposal aims to investigate the cognitive and neural processes that support the ability to integrate both languages during speaking and listening. In so doing, we will develop a new fMRI method that makes use a referential communication task with an experimenter-confederate. A developmental phase with English monolinguals will refine and optimize the methodological parameters and analyses of the new method. Experiment 1 will use this new conversational paradigm to test Spanish-English bilinguals who are also members of the U.S. Latino population. Bilingual subjects will complete the referential communication task with a bilingual experimenter from the same population thus setting an appropriate conversation context for spontaneous code-switching. Analyses will investigate the neural correlates of the production and comprehension of spontaneous code-switching. In Experiment 2, the same bilingual individuals will complete a cued language switching task also with fMRI. The results of Experiments 1 and 2 will be analyzed to compare the neural overlap between spontaneous code-switching and cued language switching. The development of a new neuroimaging technique that focuses on spontaneous conversation in bilinguals will significantly contribute to the methodological tools available to language researchers generally, and more importantly, it will broaden the research focus to minority language and dialect speech acts that have been underrepresented in the language sciences.

A central goal of cognitive science is to understand how the human brain acquires language and uses it to communicate with others. Basic research into understanding human language abilities also holds practical value in areas such as education (where difficulties with language pose a significant barrier for academic and professional success), engagement with social media (where new forms of communication are rapidly developing), and artificial intelligence (where machine learning and other algorithmic methods endow computers with increasingly sophisticated human-language communication). Traditionally, basic research on human language has taken a divide-and-conquer approach, with one part of the research community focused on how children learn languages and the other focused on how adults dynamically comprehend and produce language in real-time. However, exciting new research has bridged this otherwise artificial research divide. Neural and eye-tracking methods have revealed that infants deploy their incomplete understanding of language in real-time, almost as quickly as their adult expert counterparts. Likewise, adults have been found, under specific conditions, to be highly adaptive to language, learning new patterns of speech, new words, and even new syntax from brief exposure. This proposal highlights this new interdisciplinary work and resulting breakthroughs.

The funding will support a special session entitled ?Language Acquisition and Language Processing: Finding New Connections? to be held at the 34th Annual CUNY Conference on Human Sentence Processing (March 4-6, 2021), hosted at the University of Pennsylvania in Philadelphia. The session will include presentations from six eminent researchers who bridge language learning and processing, a poster session dedicated to this theme, and travel awards for young student investigators. By bringing researchers studying language acquisition to the premier conference on adult language processing, the session will permit the cross-fertilization of ideas, spawning new important topics of study within human language.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.