Karen Emmorey - US grants

To linguistically represent and encode the visual world (i.e., to talk about what we see) requires a crucial interface between language and spatial cognition. Signed languages can provide unique insight into this interface because physical space is used directly to encode linguistic and spatial relations. American Sign Language (ASL) differs from spoken languages in that it is expressed with the hands in space and perceived by the visual system. Thus, ASL also presents an opportunity to explore how the modality in which a language is conveyed affects the nature of grammatical encoding (e.g., how languages express different linguistic relations) and the nature of language processing (how people interpret and produce language in real time). Currently, very little is known about language processing in ASL, and the proposed studies will be some of the first to investigate how a visual language is perceived and understood on-line. The studies explore the ramifications of using space for expressing linguistic contrasts in referential and spatial domains. Specifically, the experiments focus on the use of space to express coreference relations, frames of reference, and spatial information. The major goal of the research is to explore the grammaticization of space in sign language and its impact on language processing. The project contains two primary lines of inquiry: 1) on-line tracking of spatial anaphora and 2) the use of space to represent space. The first group of experiments explore how signers understand and maintain the association between referents and their spatial loci during real time processing. The results of these experiments will provide insight into the nature of the processing mechanisms required for sign language and will help determine what aspects of referential processing hold across language modalities. The second line of experiments investigate the consequences of using space to represent space by investigating how deaf signers create spatia l mental models from signed discourse, how signers chose a spatial reference frame, and how signers integrate physical signing space with a mental representation of space. These experiments will illuminate how physical space is manipulated linguistically to communicate locative information and will clarify how such spatial structure is interpreted during processing. Together, these results will elucidate the nature of the interface between language and spatial cognition.

The study of signed languages offers a window into how the modality of language transmission and perception affects the nature of linguistic structures. The focus of this workshop is the grammatical structure and cross-linguistic variation of classifier constructions in sign languages (a.k.a. classifier predicates, verbs of motion and location, and polymorphemic verbs). Early research suggested that these forms can be analyzed as combinations of discrete morphemes, specifically, as predicates consisting of one or more movement roots along with several other morphemes encoding, for example, the shape or semantic class of object involved (indicated by a classifier handshape), the location, and the orientation of the object. However, several critical questions have arisen regarding the syntactic, morphological, and phonological analysis of these forms. Some researchers have suggested that these forms do not actually involve classifiers in the usual sense of the term, may involve gestural (rather than morphemic) components, and may be unique to signed languages.

In an effort to resolve the current controversy, the workshop will bring together sign linguists working on various unrelated languages for the purposes of documenting and describing similarities and differences in the grammatical structure and semantics of classifier constructions in sign languages. In addition, the meeting will bring together both sign and spoken language linguists, along with experts on gesture, to discuss and analyze the grammatical and typological nature of these constructions. An essential objective of the workshop is to produce a volume containing the presented papers (with revisions after workshop discussions). Such a book will be valuable for several reasons: 1) it will establish a standard for the field of sign linguistics with respect to current analysis and debate regarding classifier constructions, 2) it will provide spoken language linguists with broader insight into the nature of classifier systems in human languages and 3) it will add to our general understanding about the relation between language and gesture, the nature of grammaticization, and the effects of language modality on linguistic structure.

DESCRIPTION (provided by applicant): Bilingualism provides a valuable tool for understanding language processing and its underlying neurocognitive mechanisms. However, the vast majority of bilingual research has involved spoken languages. Speech-sign bilinguals or bimodal bilinguals use separate perceptual and motoric systems for each language. This separation of systems affords the unique possibility of producing and perceiving elements from two languages at the same time, a phenomenon called code-blending. In contrast, speech-speech or unimodal bilinguals are restricted to producing one lexical item at a time because both languages rely on the same output channel (the vocal tract). This project investigates the implications of bimodal bilingualism for models of language production, for how bilingualism affects cognition, and for the brain's ability to manage and adapt to two language systems. In a series of psycholinguistic experiments, we investigate the consequences of dual- language activation for fluent and failed lexical access and explore processing effects specific to bimodal bilinguals (e.g., code-bends, sign language iconicity, changes in co-speech gesture, and the interpretation of facial gestures). A second series of studies investigates how bilingualism leads to changes in cognitive ability. The need to continuously control two languages during speech processing has been linked to advantages in cognitive control for unimodal bilinguals. Bimodal bilinguals provide the opportunity to consider whether these advantages arise from knowledge of two linguistic systems or from competition for a single modality (speech). We examine whether bimodal bilinguals exhibit particular cognitive advantages in spatial working memory and cross-modal monitoring. A third series of studies investigates whether the psycholinguistic and cognitive effects that we observe in behavior give rise to specific changes in the functional and structural architecture of the brain in both unimodal and bimodal bilinguals and whether these neural changes are tied to early brain development. Together these studies will characterize properties unique to bimodal bilinguals, while also revealing how bilinguals manage the activation of two languages in a single cognitive system. More generally, the results will enhance our understanding of the linguistic, cognitive, and neural systems that support language processing in all speakers. PUBLIC HEALTH RELEVANCE: With the spread of bilingual approaches to deaf education and the passage of the Americans with Disabilities Act (which has resulted a great need for sign language interpreters), more and more people are becoming bilingual in American Sign Language (ASL) and English. This project documents the linguistic, cognitive, and neural consequences of such "bimodal" bilingualism. The findings will help clinicians gain a more complete and accurate assessment of cognition and language in these bilinguals, will inform education and health matters related to bilingualism in general, and will be important in designing appropriate interpreter training programs, which will benefit deaf individuals and their families.

For over a century educators and researchers have been trying to determine how profoundly deaf children learn to read. Deaf children are in the unique position of learning to read and write a language that they do not speak and cannot hear. Unlike people who can hear, deaf people experience English orthography in two forms: as printed text and as fingerspelling, in which each alphabetic letter is represented by a distinct hand configuration.

This project investigates one possible route that deaf readers may utilize to develop mappings between letters and sounds: the fingerspelling system of American Sign Language (ASL). By identifying the similarities and differences between reading print and "reading" fingerspelling, one can identify how fingerspelling might be most effectively used in reading instruction. Another aim of the project is to use functional Magnetic Resonance Imaging (fMRI) to investigate the brain areas that support the reading of print and fingerspelling by deaf readers. Of particular interest is whether the so-called "Visual Word Form Area" is activated for fingerspelling and whether deaf readers use the same neural circuits for reading as people who can hear. This project will compare the brain areas that are activated when reading words for deaf adults, for hearing adults who do not know ASL, and for hearing people who are bilingual in ASL and English. Finally, a parallel aim of the project is to increase the representation of deaf people in science by including deaf research assistants on the project and providing an accessible environment for deaf students to gain research experience.

This collaborative project will record and study the properties of lexical forms in American Sign Language. Almost everything we know about human language comes from the study of spoken languages. However, only by studying sign languages is it possible to discover which linguistic rules and constraints are universal to all human languages and which depend on the particular properties of an individual language. By studying sign languages researchers can uncover language patterns that are tied to the nature of the articulators (i.e., the hands vs. the vocal tract) or that are linked to the specific way a language is perceived (i.e., visually vs. auditorally). Researchers can also uncover language patterns that result from properties that systematically vary between spoken and signed languages, such as the high prevalence of iconic forms (words that resemble what they mean) in sign languages. Psychological and linguistic research on spoken languages has relied on lexical databases--repositories of information about the words of a language--to identify factors that influence how words are comprehended and produced, to understand how words are organized and structured in the mind and brain (in our "mental lexicon"), and to discover the linguistic patterns that are present in languages. Unfortunately however, there is currently no comparably large lexical database for American Sign Language (ASL), the sign language used by deaf and hearing people in the United States.

A primary aim of this project is to create a large, searchable, and publically available database of approximately 2,500 ASL signs. The database (called ASL-LEX) will contain the following information for each sign: subjective frequency-of-use ratings, iconicity ratings from both deaf signers and hearing non-signers, sign duration measures, lexical category information (e.g., noun, verb, etc.), and codes for sign-based phonological features (e.g., location, handshape, movement) that can be used to calculate whether the form of a sign is relatively common (has many form 'neighbors') or relatively unique (has few 'neighbors'). A second aim is to use ASL-LEX to conduct the first quantitative analysis of the ASL lexicon in order to uncover regularities in the way that phonological features appear (or do not appear) in ASL signs and how these patterns are influenced by sign properties such as frequency and iconicity. A third aim is to conduct experiments to determine the psychological reality of these phonological patterns (e.g., do signers unconsciously know which patterns are common and which are rare?) and to discover how phonological and lexical properties impact how quickly a sign is recognized (using a novel sign recognition technique) and produced (using a picture-naming task). Data from these experiments and related materials (e.g. picture stimuli) will be made available to the public through ASL-LEX. These materials constitute essential tools that will allow scientists and educators to create well-controlled ASL stimuli for use in research and the classroom. ASL-LEX can also be used by educators and early intervention specialists to develop benchmarks for assessing vocabulary development in signing children, (e.g., do children know the most frequent signs?) and to support literacy development (e.g., to find sign-based "rhymes"). A parallel aim of the project is to increase the representation of deaf people in science by including deaf researchers on the project and by providing an accessible environment for deaf students to gain training and research experience.

The advent of non-invasive brain imaging at high spatial and temporal resolution in awake, behaving human beings has had a profound impact on the study of language in the brain. It is now the ten-year anniversary of the first international conference on the Neurobiology of Language, a field that has grown substantially since then and continues to blossom. Studying the neurobiology of language requires highly specialized skills (e.g., brain imaging) and broad multidisciplinary knowledge (e.g., psychology, linguistics, neuroscience). In order to promote scientific inquiry of the highest quality, it is vital to promote dialogue and interaction among the relevant disciplines. This project explores how to facilitate this interaction.

The "neurobiology of language" is the biological implementation for representations and processes involved in the production and understanding of speech, sign, and language in context. Previously, brain structures and functions for language could only be inferred from studying humans with brain injury. By monitoring the anatomical encodings and physiological dynamics of language processing in awake, behaving human beings, researchers can now study human language at a biological precision previously possible only in animal models (of functions other than language). Concomitant with these new approaches has been a dramatic increase in the number of investigators dedicating their effort to understanding the biological basis of language. The growth in the number of investigators and the use of expanded techniques and theoretical perspectives have driven the need for enhanced transparency, integrity, and robust and rapid research dissemination. The PIs will explore these topics while developing systems and operations for open science, facilitating reproducibility and replicability of research, and dissemination of new findings.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Being able to read efficiently is critical to success in modern society. Populations with low literacy rates (e.g., individuals with dyslexia and deaf individuals) suffer from a number of negative outcomes, including poor academic and employment opportunities which can lead to reduced upward economic mobility, limited access to public information which can lead to increased fatality and health risks, as well as reduced civic engagement which can lead to increased feelings of isolation and depression. In order to improve literacy rates in these populations, researchers first must understand the mechanisms underlying successful reading so that interventions will be appropriately targeted. Decades of literacy research and instruction have focused on the importance of language abilities (particularly vocabulary and the ability to sound out words) while largely ignoring the role of attention, perception of text in non-central (peripheral) vision, and the coordination of these processes to support efficient reading. The investigators propose that visual and linguistic information obtained from the text in peripheral vision both play an important role in reading efficiency, but that they do so in different ways. For example, individuals with dyslexia exhibit abnormal eye movement behavior when reading text, but not when moving their eyes between simple visual elements such as dots. This suggests that reading difficulties arise less from needing to move the eyes precisely than from the requirement to process text linguistically. Moreover, some deaf individuals are more efficient readers than their hearing counterparts, suggesting that their reading behavior may be augmented by either auditory deprivation (which may enhance visual processing abilities) or experience with American Sign Language (ASL; which may enhance linguistic processing abilities). However, little is known about the degree to which deaf readers’ abilities are linked to their unique visual or linguistic experiences. By clarifying these mechanisms, this project has the potential to inform deaf education policy by increasing scientific understanding of the role of sign language in the cognitive development and academic success of deaf individuals. By extension, the investigators’ work will also be useful for understanding dyslexia by revealing the distinct contributions of visual and linguistic processing to reading success that have been implicated as potential deficits that underlie dyslexia.

This project investigates the distinct contributions of non-linguistic visual processing and word recognition abilities to reading success in both hearing and deaf adults. The investigators use gaze-contingent eye tracking reading paradigms to gather reading efficiency measures (e.g., reading rate, and fine-grained eye tracking metrics) at the participant level and conduct regression analyses to investigate what abilities predict these aspects of reading proficiency. To assess the predictors of reading ability, the investigators collect measures of general cognitive and linguistic abilities (e.g., English text comprehension, English spelling ability, English vocabulary, non-verbal IQ, and ASL comprehension), as well as conduct experiments that assess each individual’s ability to process various types of information in peripheral vision (e.g., simple visual features, English words, ASL signs), and to make eye movements to peripheral locations. By accounting for both visual and linguistic processing abilities simultaneously, and by comparing reading behavior in deaf signers and reading-matched hearing adults, the investigators are able to identify how language and/or sensory experiences impact reading processes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.