Gabriel Kreiman, M.Sc., Ph.D. - US grants

Description (provided by applicant): Visual object recognition is crucial for most everyday tasks including face identification, reading and navigation. In spite of the massive increase in computational power over the last two decades, a 3-year-old still outperforms the most sophisticated algorithms even in simple recognition tasks. Understanding the computations performed by the human visual system to recognize objects will have profound implications not only to understand the functions (and malfunction) of the cerebral cortex but also for developing visual prosthetic devices for the visually impaired. We combine neurophysiology, electrical stimulation and tools from machine learning to further our understanding of the neuronal circuits, algorithms and computations performed by the human visual system to perform visual pattern recognition. In the vast majority of visually impaired or blind people, the problems originate at the level of the retina while the visual cortex remains unimpaired. Our proposal constitutes a proof- of-principle approach towards developing visual prosthetic devices that rely on electrical stimulation of visual cortex. The specific aims of this proposal are designed to test the possibility of decoding and recoding information in visual cortex: (1) Read-out of visual information from human visual cortex on line (2) Write-in of visual information in human visual cortex. We take advantage of a rare opportunity to study the human brain at high spatial and temporal resolution by studying patients who have electrodes implanted for clinical reasons. Our electrophysiological recordings provide us with a unique view of the human temporal lobe circuitry and allow us to test the feasibility of cortical visual prosthetics in behaving human subjects. PUBLIC HEALTH RELEVANCE: Towards cortical visual prosthetics one of the key challenges for the visually impaired and blind people is the lack of visual object recognition capabilities. Visual recognition is crucial for most everyday tasks including navigation and face identification. Our proposal is a proof-of-principle approach towards the development of visual prosthetics devices based on electrical stimulation in visual cortex.

DESCRIPTION (Provided by the applicant) Abstract: The brain, a physical system composed of neurons and synapses, can give rise to what seems to be the least physical property of all: consciousness. How this transformation takes place has preoccupied generations of clinicians and scientists. Advances in Neuroscience over the last several decades make it possible to enquire into the neural circuits responsible for consciousness. This proposal focuses on one particular aspect of conscious experience: the neuronal mechanisms and circuits that underlie visual awareness. We propose to study visual awareness using a combination of neurophysiology, psychophysics and electrical stimulation. We take advantage of a rare opportunity to examine activity in the human occipital and temporal lobes using neurophysiology at high spatial and temporal resolution (neurons and milliseconds) while subjects report their perceptions. We propose two experiments where visual perception is dissociated from the visual input: binocular rivalry and motion-induced blindness. In both cases, perception changes in spite of a constant visual input. We investigate where, when and how neuronal responses along ventral visual cortex (from primary visual cortex to inferior temporal cortex) change their activity patterns with the perceptual alternations. Furthermore, we ask whether those neurophysiological responses are sufficient to elicit perception by electrically stimulating local circuits. Impairments in conscious processing can be devastating and are at the core of seemingly diverse conditions as epilepsy, vegetative coma, schizophrenia and autism. Furthering our understanding of the link between brains and minds in the context of vision will pave the way for addressing other aspects of consciousness and may have profound implications in changing how we think about and address these challenging disorders. Public Health Relevance: The brain, a physical system composed of neurons and synapses, can give rise to what seems to be the least physical property of all: consciousness. Understanding the neuronal mechanisms that give rise to perceptions, feelings and thoughts is arguably one of the biggest scientific challenges today. Advances towards describing the link between neuronal activity and perception will have a profound impact in our society, particularly in clinical applications. There are multiple clinical conditions that involve altered states of consciousness including epilepsy, autism, vegetative coma and schizophrenia among others. Subjects afflicted by these conditions can live for years, yet their quality of life (and that of their surrounding friends and families) can be severely impaired. These conditions lead to social and learning challenges that often impact the afflicted individuals for life. At the intersection of these conditions resides a fundamental challenge to express conscious thoughts and to perceive conscious feelings. In all these cases, consciousness can be behaviorally defined. However, what, when and how brain activity leads to conscious experience is not understood. Here we specifically focus on visual awareness because it is easier to manipulate the visual input, we understand more about the underlying circuitry and we can quantitatively assess perception rather accurately. Our research efforts will investigate the types of neurons, locations, circuits, interactions and dynamics that give rise to visual perception.

Human beings can recognize objects in a highly selective, robust and fast manner. One of the remarkable properties of our recognition machinery is the possibility to selectively recognize objects even after transformations such as rotation, translation, scaling, occlusion and clutter. How the human brain accomplishes recognition is not well understood. More is known about processing of sensory information from receptors to the initial stages in cortex than about the subsequent transformation of perceptual data into cognition. In parallel, over the last decades, major progress has been made in building ever more accurate and sophisticated computers and devices to capture sensory information but progress has been slower in terms of developing algorithms and hardware to automatically interpret the sensory data. With support from the National Science Foundation, Dr. Gabriel Krieman is undertaking research whose aim is to elucidate the computational steps and algorithms implemented by the cerebral cortex to transform incoming inputs into cognitive functions and behavior. The research focuses on one particular aspect of cognitive experience, the neuronal mechanisms and circuits that underlie visual processing. While vision is only one of many aspects of cognition, lessons learnt from studying visual cortex can also eventually help describe other aspects of cortical function and can pave the way for research on other challenging aspects of cognition. To investigate visual cognition, Dr. Kreiman takes advantage of a rare opportunity to both stimulate and record electrical activity at high spatial and temporal resolution directly from the human brain in epilepsy patients. The study investigates tasks where visual cognition is dissociated from the incoming sensory processing in order to isolate the cognitive operations involved in recognition. The discoveries about the function of biological neural circuits will be applied to develop biophysically-inspired robust machine vision algorithms.

Visual recognition is essential for most everyday tasks including navigating, reading, and identifying objects, faces and emotions. By furthering our understanding of the transformation of perceptual information into cognition, the study is contributing to two broad goals: (1) Helping to alleviate the challenging conditions that involve cognitive disorders through the development of brain-machine interfaces; and (2) Applying knowledge about neuronal circuits to develop computational algorithms to extract cognitive information from sensory data. Building a fast, robust and reliable artificial vision system would have profound repercussions in many areas of science and engineering including pattern recognition, surveillance and security, automatic navigation, clinical image analysis and others. These scientific and engineering advances could in turn translate into important real-world applications of interest for industrial partnerships. Dr. Kreiman pursues these goals by studying the best possible system that can solve visual recognition challenges, the human brain. Understanding the visual system relies on many skills ranging from computer science to engineering to physics to neuroscience to psychology. The project serves well to train a generation of multidisciplinary students who can build on the fundamental science knowledge and apply this knowledge to challenging biological problems.

Visual cognition is orchestrated by the interaction of 'bottom-up' (feed-forward) processes that carry sensory information and 'top-down' (feed-back) processes that modulate the incoming input in the context of goals, tasks, emotions and stored information. At the anatomical level, each area within the cerebral cortex is heavily innervated by both feed-forward signals and feed-back signals. With funding from the National Science Foundation, Gabriel Kreiman, Ph.D. of Children's Hospital Corporation (Boston, Massachusetts) in collaboration with Andreas Schulze-Bonhage, Ph.D., of the Freiburg University Hospital (Freiburg, Germany), is investigating the dynamical integration of bottom-up and top-down neural signals, by combining computational models and machine learning techniques for data analysis with high-resolution neurophysiological recordings from the human temporal lobe. Researchers have long recognized that top-down and bottom-up signals play a key role in visual recognition, however, the relative contribution and interactions between these signals remain unclear. The research project is focused on a particular aspect of cognition, namely our ability to visually recognize patterns, which is central to most everyday tasks. Even the best machine computational models available today only provide a coarse approximation to the complex neurophysiological responses found in higher visual cortex. Not surprisingly, a three-year-old can outperform sophisticated computational algorithms in recognition tasks, such as navigation in complex environments or recognizing objects in cluttered scenes. The research project focuses on three progressively more complex tasks that rely increasingly on top-down influences. The first research aim involves top-down influences during recognition of objects in a cluttered visual stimulus. The second aim examines whether neurophysiological responses in the human temporal lobe can support recognition from partial object information. This question is being approached through studying the phenomenon of object completion. The third aim combines visual stimulus clutter and occlusion in a complex realistic recognition scenario. For this aim, the researchers are examining the influences of attention and task-related goals on neurophysiological activity while epilepsy patients play a custom-designed video game. These neurophysiological data take advantage of the rare opportunity to combine high-resolution neurophysiology, computational models, and behaviorally complex tasks to carry out research that would be difficult with non-human animals.

By furthering the understanding of the transformation of perceptual information into cognition, the researchers are contributing to two broader goals: The goal to help alleviate the challenging conditions involved in cognitive disorders through the development of interfaces between brains and machines, and the goal to apply knowledge about neuronal circuits to develop computational algorithms that automatically extract cognitive information from sensory data. Building a fast, robust, and reliable artificial vision system would have profound repercussions in many areas of science and engineering, including pattern recognition, surveillance and security, automatic navigation and clinical image analysis. These scientific and engineering advances could in turn translate into important real-world applications of interest for industrial partnerships. Understanding the visual system relies on many skills ranging from computer science to physics, neuroscience, and psychology. The research efforts are complemented by educational and outreach initiatives aimed at training interdisciplinary scientists. The training is producing multidisciplinary students who can build on their fundamental scientific skills and apply this knowledge to challenging clinical and engineering problems. This project is jointly funded by the Cognitive Neuroscience Program, the Social Behavioral and Economics Division, Collaborative Research in Computational Neuroscience, and the Office of International Science and Engineering. A companion project is being funded by the German Ministry of Education and Research (BMBF).

DESCRIPTION (provided by applicant): Interictal discharge events: from physiological effects to cognitive consequences. Many epileptic subjects are more likely to show learning disabilities, perform less well than peers at school and have a lower distribution of IQ scores. Additionally, many epileptic subjects suffer from important co-morbidities that often lead to learning impairments including autistic spectrum disorder, mood disorders, sleep disorders and psychosis. Recently, in addition to seizure events themselves, attention has been devoted to the interictal discharge events prevalent in a large fraction of epileptic subjects. Between seizure events, intracranial field potential recordings in epilepsy subjects are often characterized by internal stimulation events denominated interictal discharge events (IIDEs). Our proposal aims to understand how the neuronal circuits in the medial temporal lobe (MTL) responsible for learning are affected by clinical and subclinical manifestations of epilepsy. Converging evidence from neurology, electrophysiology, imaging, electrical stimulation and molecular studies point to the prominent role of the hippocampus and related MTL structures in learning. Our proposal is based on a collaborative effort between four PIs with extensive experience in working with epileptic patients (Kreiman, Ph.D., Anderson, M.D., Ph.D., Loddenkemper, M.D. and Madsen, M.D.). We study patients with intractable epilepsy implanted with electrodes to localize epileptogenic areas for resection. Preventing adverse effects in this procedure requires understanding the physiological and behavioral functions of the epileptogenic areas. Thus, in addition to addressing key questions about learning, our work may directly benefit those patients undergoing surgical treatment. Our electrophysiological recordings provide us with a unique opportunity to examine the human MTL circuitry at high spatial and temporal resolution. Although it has been proposed that IIDEs may play an important role in the cognitive deficits observed in several epileptic patients, the link between IIDEs and cognition remains only poorly understood. The goal of this proposal is to further our understanding of the physiological changes evoked by IIDEs and their consequent roles in cognitive function. We hypothesize, with others, that IIDEs in medial temporal lobe areas including the hippocampus and surrounding structures play an important role in the cognitive impairments prevalent in epileptic subjects. We will test the hypothesis that IIDEs influence learning performance and physiological responses in the temporal lobe. We will further assess the spatial and temporal specificity of potential modulatory or disruptive effects by IIDEs. The goals of this proposal aim to test this hypothesis by combining physiological, computational and behavioral tools. The specific aims are: 2.1 Specific Aim 1: Characterize the effects of interictal discharge events on physiological activity 2.2 Specific Aim 2: Evaluate the effects of interictal discharge events on cognition PUBLIC HEALTH RELEVANCE: Interictal discharge events: from physiological effects to cognitive consequences Children with epilepsy show learning disabilities and perform less well than peers at school. The prevention and treatment of the co-morbidities associated with epilepsy is an urgent theme of brain research. We aim to understand how clinical and subclinical manifestations of epilepsy affect the neuronal circuits responsible for learning and memory formation. Our team includes neurologists, neurosurgeons and cognitive neuroscientists and combines physiological, computational and cognitive techniques to help address the cognitive challenges in subjects with epilepsy.

Today's AI technologies, such as Watson, Siri and MobilEye, are impressive yet still confined to a single domain or task. Imagine how truly intelligent systems --- systems that actually understand their world --- could change our world. The work of scientists and engineers could be amplified to help solve the world's most pressing technical problems. Education, healthcare and manufacturing could be transformed. Mental health could be understood on a deeper level, leading in turn to more effective treatments of brain disorders. These accomplishments will take decades. The proposed Center for Brains, Minds, and Machines (CBMM) will enable the kind of research needed to ultimately achieve such ambitious goals. The vision of the Center is of a world where intelligence, and how it emerges from brain activity, is truly understood. A successful research plan for realizing this vision requires four main areas of inquiry and integrated work across all four guided by a unifying theoretical foundation. First, understanding intelligence requires discovering how it develops from the interplay of learning and innate structure. Second, understanding the physical machinery of intelligence requires analyzing brains across multiple levels of analysis, from neural circuits to large-scale brain architecture. Third, intelligence goes beyond the narrow expertise of chess or Jeopardy-playing computers, bridging several domains including vision, planning, action, social interactions, and language. Finally, intelligence emerges from the interactions among individuals ? it is the product of social interactions. Therefore, the research of the Center engages four major research thrusts (Reverse Engineering the Infant Mind, Neuronal Circuits Underlying Intelligence, Integrating Intelligence, and Social Intelligence) with interlocking teams and working groups, and a common theoretical, mathematical, and computational platform (Enabling Theory).

The intellectual merit of the Center is its focus on elucidating the mechanisms and architecture of intelligence in the most intelligent system known: the human brain. Success in this project will ultimately enable us to understand ourselves better, to produce smarter machines, and perhaps even to make ourselves smarter. The Center's potential legacy of a deep understanding of intelligence, and the ability to engineer it, is tantalizing and timeless. It includes the creation of a community of researchers by programs such as an intensive summer school, technical workshops and online courses that will train the next generation of scientists and engineers in an emerging new field -- the Science and Engineering of Intelligence. This new field will catalyze continuing progress in and cross-fertilization between computer science, math and statistics, robotics, neuroscience, and cognitive science. Sitting between science and engineering, it will attract growing interest from the best students at all levels. The broader impact of the Center program could be to revolutionize K-12, and also 0-K, and 12-life with a deeper understanding of the process of learning. The ability to build more human-like intelligence in machines will transform our productivity, enabling robots to care for the aged, drive our cars, and help with small-business manufacturing. The Center team is composed of over 23 investigators, many having already made significant accomplishments in multiple research areas relevant to the science and the technology of intelligence. The Center team has a mix of junior and senior researchers, bringing expertise in Computer Science, Neuroscience, Cognitive Science and Mathematics. The institutional partners include nine institutions (MIT, Harvard, Cornell, Rockefeller, UCLA, Stanford, The Allen Institute, Wellesley, Howard, Hunter and the University of Puerto Rico), three of which have significant underrepresented student populations. The academic institutions are complemented by the Center's industrial partners (Microsoft, IBM, Google, DeepMind, Orcam, MobilEye, Willow Garage, RethinkRobotics, Boston Dynamics) and by world-renowned researchers at international institutions (Max Planck Institute, The Weizmann Institute, Italian Institute of Technology, The Hebrew University).

Learning and memory formation play a central role in our everyday lives. The critical nature of our ability to acquire new information and to remember previous events is manifested in a variety of conditions that affect memory ranging from learning deficits in children all the way to Alzheimer's disease and related conditions in the elderly. Previous studies have shown that the hippocampus and related structures in the medial temporal lobe (MTL) play a critical role in memory formation for places, images or words. Less is known about how spatiotemporal sequences and events are encoded and recalled. By using movies as a proxy to real-life memory formation for episodic events. Dr. Gabriel Kreiman at the Children's Hospital, Harvard Medical School, will investigate how neuronal activity in the human MTL support the formation of episodic memories during movie events and how the hippocampus interacts with neocortical areas and the amygdala during learning. Computer vision and machine learning techniques will be used to develop detailed annotations of movie events to characterize each sequence of frames in terms of its low-level properties, high-level content and emotional aspects and to predict memory formation for episodic events from audiovisual and emotional content. Furthermore, the researchers will combine this machine learning approach with neurophysiological recordings in the MTL of patients with epilepsy to understand how episodic memories for short movie events are encoded and recalled.

Advancing our understanding of the neural circuits and interactions that orchestrate memory formation will pave the way towards developing innovative technologies to directly interact with the human MTL to enhance learning of everyday episodic information. The machine learning approach used here to characterize recall of complex audio-visual events will provide systematic metrics to make predictions about memory formation, therefore holding the promise of predicting recall in educational settings. Studying how hippocampal neurons participate in the encoding and recall of episodic events is likely to have major clinical implications for understanding and developing interventions for neurological conditions including age-related dementias and learning disabilities in children. This project will also offer valuable training opportunities for undergraduate students, graduate students and postdoctoral fellows to gain first hand experience in a cutting edge interdisciplinary study involving computer science, cognitive science, neurophysiology and neurology.

? DESCRIPTION (provided by applicant): During the last decade, a plethora of novel transcripts has been uncovered, many of which come from regions formerly considered to constitute junk DNA. The characterization of those RNA species during development and disease has led to a burgeoning field within the biology of gene expression. In this proposal, we take initial steps to define a similar path with the proteome. We develop the computational infrastructure and provide a proof-of-principle for the existence of novel peptides derived from regions of the genome that are traditionally considered to be non-coding. We use the new algorithms and tools to begin to identify and define novel peptides derived from presumed non-coding regions across different developmental conditions using the mouse brain as a model system. The framework will include software tools that will allow researchers to build custom-databases from RNA-seq experiments, making it possible to search for translation products without relying on annotation databases. We will carry out rigorous validation experiments and systematic characterization of the novel non-canonical translation events and probe function of select novel proteins. The proposed research has the potential to provide a paradigm-shift for proteomics as researchers will no longer be limited by annotated databases. Since both mass spectrometry and RNA-seq experiments are now practical and no longer cost-prohibitive for most labs, the proposed framework will be of general use and it will be important to understanding the relationship between the genome, transcriptome and the proteome across diseases and biological paradigms.

? DESCRIPTION (provided by applicant): Anxiety and paranoia can lead to a significant and debilitating impact on quality of life. Both anxiety and paranoia are believed to involve abnormalities of a threat detection system that includes limbic structures such as the amygdala and interactions with neocortical areas. This proposal aims to directly examine the spatiotemporal dynamics underlying the threat detection neural circuit by performing physiological recordings in the relevant structures within the adult human brain. We will record invasive neurophysiological data in epilepsy patients implanted with electrodes for clinical reasons while they perform cognitive tasks aimed to discriminate threat signals. By virtue of the high spatiotemporal resolution, extensive and dynamic recordings, we will be able to investigate the following fundamental questions: 1) Do threat stimuli (words and audiovisual inputs) engage the amygdala and related medial temporal structures to a greater degree than non-threatening stimuli? 2) Is there a greater degree of limbic engagement to non-threatening stimuli in subjects with elevated levels of paranoia and anxiety? 3) Is the interaction between neocortical areas (including language and visual cortex) and medial temporal structures enhanced during the processing of threat stimuli? (4) Is threat detection invariant to the specific input modalities suh as semantic or audiovisual access? To address these questions, we have assembled a team with expertise in Neurology and Psychiatry (Dr. Weisholtz, Dr. Silbersweig, Dr. Butler), epilepsy (Dr. Weisholtz, Dr. Dworetzky, Dr. Madsen) and systems and computational neuroscience (Dr. Kreiman). We will address these questions by examining localized evoked potentials and induced high- frequency activity in the brain regions of interest and by contrasting the response between threat and neutral stimuli. Interactions between regions will be explored by calculating the degree of inter-regional oscillatory phase locking and correlated gamma power. We will assess the degree of paranoia and anxiety using the Paranoia Rating Scale and the State-Trait Anxiety Inventory.These research efforts will contribute to a better understanding of the inner circuits responsible for anxiety and paranoia, and ultimately to the development of predictive tools and targeted therapies.

? DESCRIPTION (provided by applicant): Our perceptions, behaviors, decisions, feelings and thoughts depend on putting together different sources of information. The long-term goal of our project is to elucidate the computations and interactions among brain areas subserving integration of cognitive information across space and time. As a paradigmatic example of cognitive integration, we focus on the problem of visual pattern completion. We can make cognitive inferences and recognize heavily occluded objects from partial information. Cognitive integration must act over time (e.g. comparing current states with past ones), across space (e.g. evaluating signals across different parts of the visual field) and across brain areas (e.g. simultaneously considering bottom-up inputs in the context of prior knowledge). To further our understanding of the neural circuits orchestrating pattern completion, here we record intracranial field potentials from temporal and frontal cortex through electrodes implanted in epilepsy patients for clinical reasons while they perform visual recognition tasks. By virtue of the high resolution of our recordings, we aim to elucidate the circuits, brain areas and dynamic interactions across areas involved in spatiotemporal integration during pattern completion. The results from these investigations will provide initial steps to characterize and constrain the fundamental problem of how information is integrated by cortical circuits.

Humans rapidly process images and scenes so they can understand what is occurring in the world around them. Humans do this so well that they outperform existing computational models' ability to understand what elements exist in the scene, their location, or the actions they are involved in. One limitation of computational models is that they do not provide a detailed interpretation of a scene's individual components the way that humans do. For example, the computational model may successfully label an image as containing a horse, but humans will also naturally identify smaller components of the horse, such as the eyes, ears, mouth, mane, legs, tail, and so on. Identifying these individual components and their relationships is an essential part of human visual processing. Such differences in visual understanding create a challenge for constructing artificial computational systems that see and interpret the world similarly to humans. These fundamental limitations are related to the fact that existing computational systems rely primarily on what is called 'bottom-up processing', the sequential processing of visual features from simple to complex ones, which does not account for how human cognition influences meaningful recognition of the image. Our main goal is to investigate the computational principles and neurobiological systems that allow for integration of cognitive experience within visual processing. We combine psychological studies in humans, neurophysiological recordings of brain tissue, and computational work to build an integrative model capable of extracting complex meaning from images, in a way that more closely resembles human capabilities. The research will have broad implications in understanding how the brain processes images and the neural circuits that are involved. Additionally, the insights obtained from this project could have applications in a broad range of domains including robot vision, automatic navigation, surveillance, and automatic clinical image understanding. As part of the project we will establish a summer course based on the research products in which we will train the next generation of scholars at the interface of brains, minds, and machines.

In the human brain, information flows both from low to higher visual areas, as well as in the opposite direction throughout ventral visual cortex. This bi-directional processing has a fundamental role in cortical computations. Yet, the functions implemented by the top-down components form an open problem in visual cognition. Understanding the limitations of feed-forward processing (from lower to higher visual regions) will shed light on the mechanisms by which prior knowledge is integrated with bottom-up inputs, and guide development of new algorithms for extracting useful meaning from sensory input. Applications of deep network models now play a significant role in machine learning across multiple domains, but these fail to capture fundamental aspects of visual processing. Our research program examines the possibility that existing feed-forward recognition models constitute a first stage leading to the initial activation of category candidates, which is incomplete and often inaccurate. The first stage then triggers the application of class-specific processes, which recover a richer and more accurate interpretation of the visible scene, and reject initial false candidates. The proposal involves three main components: (i) Psychophysics experiments to evaluate the accuracy and speed of how humans extract meaning from images; (ii) Invasive neurophysiological recordings along the human ventral visual cortex to understand the neural circuits involved in extracting meaning from a novel data set of minimal images; (iii) A computational model that integrates bottom-up computations with top-down signals to extract meaning from images. The research efforts will be combined with educational and outreach activities aimed at disseminating the scientific insights and incorporating cutting-edge research into training opportunities for undergraduate and graduate students.

Project summary Flexibility in behavior is a critical and ubiquitous component of intelligence. The brain adapts to control our behavior in different ways depending on the specific situation. Frontal cortex plays a fundamental role in flexible and adaptive behavior but the mechanisms underlying such cognitive control are poorly understood and are often studied in the context of deciding how to interpret conflicting sets of information. In this proposal we aim to systematically investigate whether there are invariant computations underlying cognitive control across multiple different scenarios and tasks by taking advantage of a rare opportunity to invasively and directly investigate the neural signals in the human brain. Furthermore, we aim to take initial steps towards interfering with such cognitive control signals via invasive electrical stimulation of the underlying circuits.

The project will investigate how the brain processes language, one of the most consequential questions we can ask. Language skills significantly affect lifetime income and social disparities. The loss of language or a halt in its development can be devastating. At the same time, insights from the brain that could improve machines’ understanding of language opens up new applications -- from web search, to voice assistants, to, one day, robots that can help us in our daily lives. Using neuroscience to understand what happens during language use, what goes right and wrong, what linguistic structures and theories are used by the brain, would be revolutionary. To do this, neuroscientists use many of the same tools as those created for machine learning. Those machine learning tools have improved tremendously using large datasets, changing what machines are capable of; yet the neuroscience of language has been largely unable to reap these rewards. We will provide that data, the new methods and metrics, required to enable neuroscience to scale up and take advantage of modern machine learning. At the same time, scale in machine learning has democratized access to tools; scientific communities can investigate questions that pertain to them. Today, only a few groups have the resources to collect data and investigate questions around the neuroscience of language, leaving many communities in the dark. A large-scale central repository of data, tools, and benchmarks will democratize access to the study of language in the brain, one of the core aspects of what makes us human.

Our technical goal is to produce the largest dataset, by a factor of 1000, for investigating the neuroscience of language along with new types of models that exploit this data, and benchmarks which formalize linguistic questions to derive insights about the language network and the structure of language. Thus far, investigations in the neuroscience of language have only been able to provide small snapshots of the language network on different datasets, making it hard to build a coherent picture. A single large-scale dataset with precise benchmarks that formally define what hypotheses in linguistics mean in terms of neural data will enable the community to ask many questions of the same data, allowing for a synthesis of the structure and operation of the language network. At the same time, large-scale data is known to be required to probe the understanding of artificial language models. It is likely that if tens of thousands of sentences are required to probe an artificial language-neural-networks and derive meaningful insight, the same scale of data will be required per subject to probe biological language-neural-networks. Formalizing questions around benchmarks on a common dataset has resulted in astronomical progress in many fields from parsing (Penn Treebank) to image recognition (ImageNet); we will apply this same methodology to the neuroscience of language. This process is so efficient, in part, because it casts questions in a way that non-domain experts can access; machine learning experts need not concern themselves with linguistic minutia, they will be able to improve decoding of language from the brain and thereby drive insights by following existing protocols. By putting forward linguistic questions in a precisely-defined manner, we will also enable cross-disciplinary collaboration: linguists will be able to propose benchmarks which are questions around the performance of classifiers or mapping between networks and neural activity. These benchmarks will provide a common mathematical language by which different fields can express their key questions, in a way that has not been possible before because no dataset existed that could even support such work. We see a future where neuroscience, linguistics, natural language processing, and machine learning act as an integrated whole to ask the right questions about language in the brain, to develop new tools that support answering those questions, and to probe a large-scale resource that supports building a coherent picture of the language system.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Project summary Social interactions ? a ubiquitous and essential aspect of our everyday life ? rely on the correct perception of the actions of others. Experimental evidence suggests that the ability to interpret others? actions is subserved by a network of areas of which motor cortex represents an important hub. The precise role of this motor hub in action perception is still largely unknown. In this proposal, we will assess the mechanisms underlying action perception through invasive recordings of intracranial field potential signals in humans and computational analyses methods. This approach will allow us to study the role of motor processes in action perception with high spatiotemporal accuracy.