Roger Levy, Ph.D. - US grants

This CAREER award investigates how humans integrate a wide variety of
information sources to achieve rapid, accurate natural language
comprehension subject to the physical and cognitive constraints under
which it takes place. The project's primary empirical focus is on
reading, a mode of information exchange of unexceeded importance in
literate societies. Reading involves a rapid sequence of targeted eye
movements throughout a text -- recordable through modern eye-tracking
technology -- from which noisy sensory input are obtained and
integrated with prior knowledge to resolve perceptual and linguistic
uncertainty. The central goal of this project is thus to develop,
implement, and test a computational model of language comprehension
and eye movement control in reading built on principles of
probabilistic inference and rational action, using the tools of
natural language processing (NLP) technology, reinforcement learning,
and behavioral psycholinguistic experimentation.

The success of this project is likely to have major impact in the
field of human sentence processing, bringing a new level of nuance and
detail to both theory and data analysis, and will bear on broad
current debates in cognitive science regarding rationality in
cognition. Additionally, the results of this basic research project
have a wide range of potential applications ranging from intelligent
tutoring technology to language-impairment diagnosis to cognitive
ergonomics. Together with this research program, the project involves
an educational program including a new textbook on probabilistic
models in the study of language, new undergraduate and graduate
courses, and tutorials and courses on computational psycholinguistics
at major conferences and summer institutes.

This CAREER award is co-funded by two directorates:: CISE/IIS and SBE/BCS.

DESCRIPTION (provided by applicant): Language comprehension is central to our experience of the world. However, the ease with which we understand the language we hear belies the real difficulty of the task. In reading, the eyes can only obtain detailed information from a span of a few characters at once. In auditory comprehension, percepts change moment by moment, and listeners must distribute cognitive resources between attending to the current speech stream and maintaining short-term memory of earlier parts of the speech stream. Despite these challenges, comprehenders have a remarkable capacity to integrate a wide variety of information sources moment by moment as context in the sentence comprehension process. Our project focuses on one of the most important aspects of this capacity: the integration of bottom-up sensory input with top-down linguistic and extra- linguistic knowledge to guide inferences about sentence form and meaning. Existing theories have implicitly assumed a partitioning of interactivity that distinguishes the word as a fundamental level of linguistic information processing: word recognition is an evidential process whose output is nonetheless a specific winner that takes all, which is in turn the input to an evidential sentence-comprehension process. It is theoretically possible that this partition is real and is an optimal solution to the problem of language comprehension under gross architectural constraints that favor modularity. On the other hand, it is also possible that this partition has been a theoretical convenience but that, in fact, evidence at the sub-word level plays an important role in sentence processing, and that sentence-level information can in turn affect the recognition of not only words further downstream but also words that have already been encountered. In this project we use computational simulation and behavioral experimentation to explore the implications of removing this partition from theories of probabilistic human sentence comprehension. Our preliminary work demonstrates that, rather than vitiating their explanatory power, allowing interactivity between sentence-level and word-level processing may expand the scope of such theories, accounting for a number of outstanding problems for the notion of sentence comprehension as rational inference. The new work proposed for this project focuses primarily on elucidating this idea of rational sentence comprehension under uncertain input to the problem of eye-movement control during reading. The study of eye movements in reading is an ideal setting for further developing such a theory, because it is well-known that moment-by-moment inferences rapidly feed back to eye movements, but accounts of why given couplings between sentence-comprehension phenomena and eye-movement patterns are observed remain poorly developed. In this project we develop a new model of rational eye-movement control in sentence comprehension based on rich models of probabilistic linguistic knowledge, gradient representations of perceptual certainty, oculomotor constraints, and principles of optimal decision making. We propose a number of behavioral studies designed to test the foundational principles underlying this model, primarily using eye-tracking but also using other paradigms where appropriate. Finally, we propose specific methods to test the ability of the model to predict realistic eye movements in the reading of various types of texts. At the highest level, our work promises to lead us to a new level of refinement in our understanding how the two fundamental processes of word recognition and grammatical analysis-commonly understood as independent of one another-are in fact deeply intertwined, how they jointly recruit the two key information sources of sensory input and linguistic knowledge, and how they guide not only moment-by-moment understanding but even detailed patterns of eye movements in reading. This work lays the foundation for deeper understanding and improved treatment of both language disorders and age-related changes in reading and spoken language comprehension, which can arise as a consequence of processing breakdowns involving either or both of these key two information sources.

People often mean more than they say. To take an example, imagine Adam says "I could use a cup of coffee" and Bob responds by saying "There's a place called Joe's around the corner." We understand this as a coherent exchange even though Adam's utterance wasn't phrased overtly as a question and Bob didn't explicitly say that Joe's sells coffee. Extracting this rich additional meaning requires us to consider sentences in light of both the context they are used in and the cooperative motivations of Adam and Bob in using language (what are called "pragmatic inferences"). This project is devoted to constructing formal models of these pragmatic inferences. Modeling pragmatic inference is a major scientific challenge in the study of language and the human mind and a key to the future development of autonomous intelligent systems that can communicate with humans using natural language. Machines that can do robust language understanding in context will pave the way for societally beneficial technological applications such as adaptive intelligent tutoring and assistive technologies.

The technical core of the project involves developing and extending models of pragmatic reasoning, drawing on ideas and insights from decision theory, probabilistic models of cognition, bounded rationality, and linguistics. In particular, the work extends the recently developed family of "rational speech act" (RSA) models, which provides a set of formal tools that can be used to address basic challenges in psycholinguistics concerning how major principles of pragmatic inference fall out of simple assumptions about cooperativity and shared context among conversation participants. This enterprise has the potential to fill a major open theoretical gap in our scientific understanding of human language and social cognition. Project work includes developing computational Bayesian models of semantic composition and pragmatic inference and testing those models using controlled psycholinguistic experiments. The work will also yield new models and publicly available datasets and will contribute to interdisciplinary connections by creating and reinforcing links between linguistics, psychology, and computer science.

Language is the most discrete, measurable cultural record of the human mind, and is uniquely expressive among the communicative systems found in nature. Every day we comprehend hundreds of sentences that we hear or read but have never encountered before, and we produce hundreds more. Yet our success at these many acts of communication belies the difficulty of the task: language is rife with ambiguity, our attention is limited, our environments may be noisy, and we often have incomplete information about the shared knowledge and beliefs of the people we engage with. This ability, unique to our species, poses profound challenges for our scientific understanding of the capabilities of the human mind. Deepening our understanding of these capabilities requires a combination of ideas and methods from linguistics, psychology, and computer science. Advances in this area help lay the groundwork for improvements in natural language technologies such as document summarization, paraphrasing, question answering, and machine translation, and in better identification, diagnosis, and treatment of language disorders.

Within this broader research enterprise, this project focuses on the "edge of the lexicon", elucidating the conditions under which a linguistic expression begins to get stored in the mind of the native speaker who uses it, and the consequences of the expression being stored as a holistic unit. Native speakers know both productive rules that license and allow interpretation of phrases and sentences that they have never before encountered and a rich inventory of lexical items that can be combined through these productive rules. Many of these lexical items are individual words, but there is evidence that specific, frequent multi-word expressions, such as "meat and potatoes" or "large majority" may also get stored in the lexicon. This project combines artificial intelligence-based computational models, large linguistic datasets, and controlled psychological experimentation to explore the edge of the lexicon, probing how direct experience with specific multi-word expressions leads to their being stored in one's mental lexicon, how such storage is reconciled with productive knowledge in language comprehension and production, and how these expressions emerge and change over time.

Reading is the most widely practiced skill in the world. It occupies many hours of our daily lives and is crucial to functioning successfully in modern society. When we read, our eyes move over the text in a way that reflects how we perceive, process, and understand it. This project uses eye tracking to develop a new approach to studying how our eyes move during reading and what eye movement patterns can reveal about readers' linguistic knowledge and how they interact with text. In particular, the researchers develop eye tracking-based methods that automatically determine readers linguistic proficiency, how well they understand the text, and how difficult they find it. The project improves automatic text processing by machines, by taking into account information about how humans read. This research program benefits society by advancing our scientific understanding of language processing during reading and enabling new technologies that support human readers from a wide range of backgrounds and skill levels. It lays the foundations for future digital platforms that make it easier for people to access textual information, improve literacy, learn new languages, and personalize text content and complexity according to readers' needs and goals.

The project introduces a novel conceptual and computational approach to studying human reading with both native and non-native speakers, by leveraging broad coverage eye movement patterns during reading of free-form text. It develops a computational framework that connects eye movement in reading to linguistic properties of the text, the readers' linguistic knowledge and their cognitive state during reading. To realize this framework, the project first focuses on using eye movement in reading to automatically predict readers' linguistic proficiency and estimate their comprehension of specific parts of a text. These and other related tasks help to characterize how language comprehension manifests in gaze and unfolds over time, and also lead to the development of a predictive computational framework for native and non-native reading. Its second focus is on integrating data and representations from eye tracking in natural language processing, with the aim of developing applications which support and enhance human reading and language learning.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Every day we understand hundreds of sentences that we have never encountered and we produce hundreds more. This success is remarkable given the noisy environments in which language takes place, the errors speakers make, and limitations of our memory and attention. The present project develops and tests a theory of robust language understanding. The investigators combine tools of information theory, natural language processing, linguistics, and experimental psychology to provide a mathematically formalized model of human language comprehension as probabilistic inference over a “noisy channel”. The project contributes to our basic scientific understanding of human language and the human mind, while strengthening bridges between psycholinguistics and contemporary artificial intelligence research. The work has wide-ranging long-term potential to enhance our understanding of healthy cognitive performance and development in the area of language and to identify and guide treatments for developmental and acquired language disorders.

In this program of research, the investigators develop a computationally and algorithmically precise theory of how human understanding of sentences unfolds moment-by-moment. This incremental noisy-channel theory is implemented using state-of-the-art symbolic and neural network-based approaches to modeling language from artificial intelligence and natural language processing. A key component includes an account of how the distributional statistics of language shape noisy memory representations used during real-time language processing. Distinctive empirical predictions regarding robustness to errors in the linguistic input and regarding when and how the proposed mechanisms influence comprehension, allow this approach to be evaluated relative to alternative psycholinguistic theories. The predictions are tested using controlled behavioral experiments on how native speakers process and interpret linguistic input.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Humans interpret language in remarkably flexible ways. In particular, our interpretations involve not only what a speaker actually says, but also what the speaker could have said but didn't. For example, if someone says "some students passed the exam," listeners probably take this to mean that not all students passed, because the speaker could have used the more informative alternative "all students passed the exam" if that had been the case. Reasoning about these linguistic alternatives appears to play a key role in language comprehension, but it remains unclear how exactly the alternatives themselves are determined. When a speaker says something, what makes one sentence a better or worse alternative than another? How do alternatives depend on prior experience with different grammatical structures? And how might children learn the ability to reason about alternatives? This research project investigates these questions by combining linguistic theory, behavioral experiments, probabilistic modeling, and machine learning to test competing theories of alternatives. In doing so, this work has the potential to advance our understanding of language in the human mind, as well as artificial models that use language in human-like ways. The project also establishes opportunities for undergraduate researchers, and produce new code and datasets that will be publicly released to the broader scientific community.

In contrast to existing theories which suggest that alternatives are generated through operations on syntactic structures, this research evaluates the idea that alternatives capture experience-based accessibility relationships between the lexicon, grammar, and context. A planned series of behavioral experiments assess how the accessibility of alternatives affects the interpretation of event causation in English periphrastic causative constructions. In addition, model simulations investigate how knowledge of alternatives may emerge over the course of language learning, simultaneously establishing a framework for testing theories of alternatives in more ecologically valid domains.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.