Marcel A. Just - US grants

To perform well in a familiar but complex task like reading, processing resources must be appropriately allocated to each of the sub-components of reading, such as recognizing words, determining the syntactic and semantic relations, inferring the referents, and organizing the information within a schema. Not all components of reading require the same allocation of resources. For a skilled reader, word recognition requires so few resources that it is automatic. By contrast, understanding the implications of a technical paragraph requires considerable attention, and may be beyond the resources of a less skilled reader. The proposed research will determine how readers in various tasks allocate processing resources to the components of reading comprehension. The experiments will measure the effects of different attention allocations by comparing the reading in normal conditions to reading performed concurrently with another task, or reading which is focused on one particular component, as in searching for a fact or proofreading. The experiments examine the comprehension processes as they occur by monitoring the readers' eye fixations on the text and using these data as indicators of the characteristics of the underlying processes. Post-reading comprehension tests assess the knowledge that was produced by the various processes. The proposed research will determine which components of reading are relatively immune to attention shifts and proceed normally regardless of competing tasks and attention-directing instructions, which ones become disengaged when processing resources are drawn away from them, and which ones compete with each other for attentional resources. The models and methodologies developed for normal reading enable us to address questions of individual differences in reading ability, reading retardation, and dyslexia. More generally, the eye-fixation methodology is providing a powerful new tool to examine how brain mechanisms operate normally or abnormally in fundamental thought processes. The medical and educational value of these analytic tools is not just a promise for the future, but is currently available for use in the service of education and mental health.

This proposal constitutes an application for an ADAMHA Research Scientist Award (RSA). The investigations will trace ongoing mental processes by tracking eye fixations and other behavioral measures during the comprehension of sentences, examining the role of working memory in comprehension. The PI will extend his skills in computer simulation of parallel production systems and other connectionist architectures. The proposed research experimentally examines how comprehension performance changes when the demands of the task exceed the supply of storage and processing resources available in working memory. the empirical studies examine working memory constraints on four aspects of language comprehension. One series of experiments investigates the relation between syntactic modularity and working memory capacity, examining the hypothesis that a resource-demanding interaction between syntactic and semantic processing occurs only in those individuals and tasks in which the required resources are available. A second series examines whether the larger capacity of some individuals permits them to maintain multiple interpretations of a structurally or lexically ambiguous sentence. A third series investigates how the storage of information over a text distance varies with the processing demands made by the intervening text. A fourth series develops the methodology of pupillometry to index the consumption of cognitive resources during language comprehension. In addition, the experimental methodologies include the measurement of gaze locations and durations during reading, measurement of word-by-word self-paced reading times, and cross-modal priming. The methodologies are used to answer questions about the time course, content, and intensity of processing. The theory will be instantiated as a computational model, namely an activation-based production system, in which both processing and storage are fueled by activation. In this model, the total amount of activation available to the system has an upperbound that corresponds to the maximum capacity of an individual. The goal of the theory is to explain how the processing of language accommodates (or fails to accommodate) the transient computational and storage demands that occur in language comprehension, and to thereby explain the variation in comprehension among tasks and individuals. One health-related implication of this research is that the theory will explain a dimension of individual differences that potentially encompasses not just normal variation in comprehension, but also variation due to extreme age, to stress, or to trauma. Second, the research develops new methodologies with clinical potential (relating eye fixations to cognitive processes and relating pupil dilation to the consumption of cognitive resources). Third, the research develops the theoretical analysis of a psychometric instrument (the reading span test) that may prove useful for measuring age and disease-related changes in language functioning.

The proposed research will examine how the limited capacity of working memory constrains the on-line processing in language comprehension. Comprehension performance changes qualitatively and quantitatively when the resource demands of the task exceed the supply. The proposed research experimentally examines I what kinds of task demands exacerbate resource shortages, how resource shortages are accommodated by the, i system, and how individual differences in maximum capacity affect performance. The empirical studies examine working memory constraints on four aspects of language comprehension. One series of experiments investigates the relation between syntactic modularity and working memory capacity, examining the hypothesis that a resource-demanding interaction between syntactic and semantic processing occurs only in those Individuals and tasks in which the required resources are available. A second series examines whether the larger capacity of some individuals permits them to maintain multiple Interpretations of a structurally or lexically ambiguous sentence. A third series investigates how the storage of information over a text distance varies with the processing demands made by the intervening text. A fourth series develops the methodology of pupillometry to index the consumption of cognitive resources during language comprehension. In addition, the experimental methodologies include the measurement of gaze locations and durations during reading, measurement of word-by-word self- paced reading times, and cross-modal priming. The methodologies are used to answer questions about the time course, content, and intensity of processing. The theory will be instantiated as a computational model, namely an activation-based production system , in which both processing and storage are fueled by activation. In this model, the total amount of activation, available to the system has an upperbound that corresponds to the maximum capacity of an individual. The goal of the theory is to explain how the processing of language accommodates (or fails to accommodate) tile transient computational and storage demands that occur in language comprehension, and to thereby explain the variation in comprehension among tasks and individuals. One health-related implication of this research is that the theory will explain a dimension of individual differences that potentially encompasses not just normal variation in comprehension, but also variation due to extreme age, to stress, or to trauma. Second, the research develops new methodologies with clinical potential (relating eye fixations to cognitive processes and relating pupil dilation to the consumption of cognitive resources). Third, the research develops the theoretical analysis of a psychometric instrument (the reading span test) that may prove useful for measuring age and disease-related changes in language functioning.

DESCRIPTION (Applicant's Abstract): The proposed research applies a resource theory to the large-scale brain network that processes language, proposing and testing a theory of the mapping between the functional processing at the cognitive level and neuronal activity at the brain level. The research examines this mapping by varying the type and amount of demand on resources and the supply of resources (as operationalized in an existing computational model) and by assessing the associated brain activation with functional Magnetic Resonance Imaging (fMRI). The studies investigate the effect of demand and type of process on the activation associated with Wernicke's area, and the relation of the language system to the more general central executive. Instead of using brain imaging to ask "which brain areas activate?" We will ask "to what degree does each area in the network activate under different task conditions and how do they interact as a function of resource demand and supply?" From the intensity patterns of the brain activation, we will construct models of the coordination and interaction of the underlying component processes involved in sentence comprehension, building on our existing simulation model. The studies use cutting-edge technologies and methodologies, exploiting the speed and sensitivity of echo-planar fMRI as well as behavioral studies and computational modeling to develop an integrated characterization of the relation between the cognitive processing of language and the organization of the brain systems underlying language. The research provides a direct link between brain activity and functional disorders in language processing. The approach allows the assessment of brain and cognitive function after stroke-induced aphasia, and before and after therapy. The new paradigm succeeds in identifying the location of parts of the language network in presurgical patients. Moreover, there is a clear promise of this approach helping to analyze language impairments that are less specific than aphasia, such as the disconnection facets of autism, Parkinsonian impairments of language, and diffuse diseases such as Alzheimer's.

This proposal constitutes an application for an ADAMHA Research Scientist Award (RSA). The investigations will trace ongoing mental processes by tracking eye fixations and other behavioral measures during the comprehension of sentences, examining the role of working memory in comprehension. The PI will extend his skills in computer simulation of parallel production systems and other connectionist architectures. The proposed research experimentally examines how comprehension performance changes when the demands of the task exceed the supply of storage and processing resources available in working memory. the empirical studies examine working memory constraints on four aspects of language comprehension. One series of experiments investigates the relation between syntactic modularity and working memory capacity, examining the hypothesis that a resource-demanding interaction between syntactic and semantic processing occurs only in those individuals and tasks in which the required resources are available. A second series examines whether the larger capacity of some individuals permits them to maintain multiple interpretations of a structurally or lexically ambiguous sentence. A third series investigates how the storage of information over a text distance varies with the processing demands made by the intervening text. A fourth series develops the methodology of pupillometry to index the consumption of cognitive resources during language comprehension. In addition, the experimental methodologies include the measurement of gaze locations and durations during reading, measurement of word-by-word self-paced reading times, and cross-modal priming. The methodologies are used to answer questions about the time course, content, and intensity of processing. The theory will be instantiated as a computational model, namely an activation-based production system, in which both processing and storage are fueled by activation. In this model, the total amount of activation available to the system has an upperbound that corresponds to the maximum capacity of an individual. The goal of the theory is to explain how the processing of language accommodates (or fails to accommodate) the transient computational and storage demands that occur in language comprehension, and to thereby explain the variation in comprehension among tasks and individuals. One health-related implication of this research is that the theory will explain a dimension of individual differences that potentially encompasses not just normal variation in comprehension, but also variation due to extreme age, to stress, or to trauma. Second, the research develops new methodologies with clinical potential (relating eye fixations to cognitive processes and relating pupil dilation to the consumption of cognitive resources). Third, the research develops the theoretical analysis of a psychometric instrument (the reading span test) that may prove useful for measuring age and disease-related changes in language functioning.

Just
0079708

This proposal requests funding for the acquisition of a 3T MRI scanner for human and animal brain imaging for use by a large, multi-disciplinary, dual-university community of users. The instrument is based on a Signa Horizon LX platform, optimized for activation studies. The very high-field strength enhances the signal to noise ratio and contrast for fMRl applications, enables the collection of very high-resolution structural images, and it is equipped for ultra-fast echoplanar imaging (EPI) that enables images to be acquired and processed at the rate of over 10 images per sec.

The scanner would be the centerpiece of a new, inter-university Imaging Institute that brings together researchers in several disciplines at the University of Pittsburgh and Carnegie Mellon: cognitive neuroscientists who study human performance in complex environments, neuroscientists combining single-cell recording and neuroimaging with monkeys, biophysicists interested in advancing MR methods, statisticians with expertise in neuroimaging, and computer scientists interested in the analysis of large-scale databases. The basic science approach aims at an interdisciplinary synergy focused on brain imaging and theoretical integration. The university-supported Imaging Institute will provide a rich infrastructure and appropriate staffing to maximize the benefits of the scanner.

The new instrument will be optimized for assessing a wide range of cognitive processes and will provide an unparalleled opportunity to relate human and animal cognition. The new facility will provide a data rich environment in which the 34 participating investigators (and a total group of 174 potential users) can combine their expertise in biomedical engineering, cognitive psychology, computer science, education, linguistics, statistics, and neuroscience. The investigators include seasoned researchers who can link brain imaging to their established research disciplines, thereby enriching the imaging research.

There are five areas of basic science research: 1. Cognitive processing (of language, problem solving, spatial processing, motor control, and learning); 2. Monkey imaging (of network activity, maturation/learning, activity dependent contrast agents, relation to single neuron activity, and structural imaging); 3. Analysis methods including statistical analysis (noise reduction, hierarchical Bayesian assessment, motion correction) and computer science analysis (machine learning, data mining, and analyzing the content of images); 4. MRI methods development including: fast fMRI imaging, respiratory/cardiac noise reduction, reduction of susceptibility artifact, metabolic and volumetric imaging, animal RF coils and contrast agents; and MR spectroscopy of metabolites; and 5. computational modeling of cognitive function, examining how a variety of computational architectures can simultaneously account for human performance and fMRI patterns.

The facility will provide extensive training and research time (20,000 hours of scanning over the next five years) to undergraduates, graduates, postdocs and faculty, and will infuse the unfolding science into the ongoing educational mission. The training activities include the development of new graduate and undergraduate courses (using a new brain imaging computer classroom), seminars and workshops on brain imaging, a new brain imaging graduate core, and summer undergraduate research traineeships. External training will include workshops, web course materials, software tools, and functional imaging and modeling data sets. Outreach activities include courses for educators, museum shows, web programs, and technology transfer with industrial partners.

The activity would advance the new interdisciplinary science, extending and integrating the growing knowledge in this area into comprehensive theories of brain and mind.

Using machine learning and cognitive modeling to understand the fMRI-measured
brain activation underlying the representations of words and sentences

Tom M. Mitchell and Marcel A. Just

Project Abstract

A number of recent fMRI studies have reported significant and repeatable differences in fMRI brain activation when human subjects perceive pictures or words describing objects from different semantic categories (e.g., pictures or words that describe tools, buildings, or people). It is currently possible to determine with good accuracy which of several semantic categories a person is thinking about, based on their brain activation.

We propose new research that builds on these recent discoveries, and seeks to understand (1) human brain activity associated with different semantic categories of objects and actions (nouns and verbs); (2) whether the brain activity associated with semantic categories can be partitioned into more primitive semantic components (e.g., does the brain activity associated with words about tools factor into one component characterizing the tool's visual appearance and a second component characterizing the motor actions involved in using the tool?); and (3) how brain activity associated with individual words is combined into more complex patterns when reading word pairs or simple phrases and sentences.

This research involves:(1) applying machine learning algorithms to discover cortex-wide brain activation patterns associated with particular semantic domains, (2) developing a computational model of human language processing that instantiates the representational principles discovered and that makes specific, testable predictions, and (3) conducting new fMRI studies to obtain novel data about human semantic category representations.

The intellectual merit of the proposed research is multifaceted. If successful, our research will lead to new scientific insights into how the brain organizes information about meanings of words, objects, and actions. It will also lead to new methods for fMRI data analysis, especially for discovering complex temporal-spatial patterns of fMRI activation that accurately distinguish different mental states. The research will also lead toward a new paradigm for developing computational cognitive models and fitting them to empirical data obtained from fMRI and from behavioral measures.

The broader impacts of the proposed research will be amplified by specific outreach activities to several communities. In addition to publishing our scientific results in the cognitive and computational neuroscience literature, we will also actively engage this community by disseminating our new experimental fMRI data through the NSF-funded fMRI Data Center, and by documenting and publishing our new data analysis algorithms on the internet. We will proactively engage the statistical machine learning community, which has much to contribute to development of new fMRI analysis methods, and will develop and disseminate teaching materials for the undergraduate and graduate educational community,including fMRI data sets. Finally, our proposed research has potential impact on the medical research community, especially regarding the study of neurological conditions such as Alzheimer's disease, dyslexia and high-functioning autism - three areas entailing a language disturbance in which we already have active research collaborations, providing a direct conduit for transferring new scientific insights that may arise from this research.

This project proposes three series of fMRI brain imaging studies to investigate high-level conceptual processes in text comprehension, particularly the comprehension of metaphor, irony, and causality. The operating characteristics of the cortical networks that underlie such processing are multifaceted, including the specification of which cortical areas activate, the time courses of their activation, and the patterns of synchronization across areas. The brain activation characteristics will be related to the underlying cognitive processes and to measures of behavioral performance, integrating the two levels of explanation. In particular, the studies will have the following specific aims, among others: ? To characterize the brain activation during the comprehension of novel, dynamically computed metaphors, in contrast to frozen metaphors (e.g. information highway) that are retrieved from memory, and in contrast to literal sentences that have no figurative meaning. 1- To determine the role of visual and auditory imagery in figurative language comprehension. 2- To examine how prosodic cues like a sarcastic tone are processed in the auditory comprehension of irony. 3- To determine how information about the emotional state of a speaker (e.g. jocular, bitter) is used to interpret an ironical utterance. 4- To examine how the relation between two events described in a text is computed, depending on whether the events are related by physical causality, social causality, serial order, or verb-based information. 5- To determine how causal connectives like Therefore affect the processing of causal relations between events described in a text. Finally, an overarching aim is to construct a common computational model within the theoretical framework that accounts for as many of the brain activation and behavioral findings as possible, postulating as small a number of plausible underlying mechanisms as possible. The research will provide tools for assessing patients with brain damage to parts of the language system and provide a basis for future therapies.

The overriding aim of this project is to relate the major symptoms of autism to abnormalities in their neural substrates, providing a neural systems-level analysis of autism, and focusing on neural systems connectivity. The primary method will be to perform fMRI studies of several different types of thinking to obtain information about underlying brain function, and to simultaneously acquire information about the size and integrity of brain tissues. The fMRI studies will provide information about cortical activation, but also about functional connectivity or the synchronization of the activation between areas. The project has developed the beginnings of a theory proposing that autism is marked by disordered connectivity among regions, particularly affecting the connectivity between frontal areas and more posterior areas. The connectivity framework is being used to formulate and test hypotheses and then integrate the new findings into a coherent theory. The new studies will deepen and broaden the understanding of the underlying neural disorder. The deepening will include the integration of several imaging modalities, so that the connectivity can be understood at more levels and in more detail. The broadening will consist of examining a wider range of tasks, to include conceptual comprehension, high-level perceptual tasks, and social tasks. This broadening is essential for determining the generality or specificity of the brain function characteristics of autism. The specifics aims are 1: To characterize brain function in the processing of higher conceptual levels of language comprehension, visual cognition, and dynamic social cognition;2: To characterize the semantic representation of single words in individual participants with autism applying innovative machine learning techniques to fMRI data;3: To relate functional characteristics of the brain to anatomical characteristics at the individual participant level;4: To further develop the theory of disordered connectivity in autism, integrating functional (fMRI) and anatomical (MRI and DTI)characteristics of autism and using computational modeling as a theory-building tool. This research will continue to build an understanding of the relationship between the behavioral impairments that characterize autism and their neural basis. The connectivity model for autism advanced on the basis of this research has significantly influenced the field and validates the complex information processing development by Minshew, Goldstein and Williams. The combined model is proving to be promising in altering conceptualizations of autism and opening the way for new intervention methods(see deception learning for example). This research addresses Autism Research Matrix Goals # 2,6,22,23,26,34.

This project seeks to develop a new understanding of how the brain represents and manipulates meaning, by bringing together the perspectives of brain imaging, machine learning and computational modeling, using converging approaches from behavioral psychology, linguistics, computer science and neuroscience. In particular, the brain activity that encodes the meanings of words, phrases and sentences is studied, along with how the brain encodes the meaning of individual words in terms of their component semantic features, how it modifies its encoding of an individual word when it occurs within a phrase or clause, and how it constructs the encoding of a phrase or clause from the encodings of its component words. This work builds on recent research showing (1) that repeatable patterns of fMRI activation are associated with viewing nouns describing concrete objects such as "hammer" or "toe," (2) that the neural patterns that encode the meanings of these words are similar across different people, and (3) that these encodings are similar whether the person views a word or a picture of the object. Whereas previous work has focused on the neural representation of single words in isolation, this project studies multiple word phrases and sentences, which comprise larger units of knowledge; for example how the neural encoding of a noun is influenced by its adjective (e.g., "fast rabbit" vs. "cuddly rabbit") and how the neural encoding of a proposition is related to the encodings of its component words (how "cut" and "surgeons" combine in the proposition "surgeons cut"). To address these questions, computational models are developed using a diverse set of training data including fMRI data, data from a trillion-word corpus of text that represents typical language use, and behavioral data from language comprehension and judgment tasks, as well as online linguistic knowledge bases such as VerbNet, and theoretical proposals from the cognitive neuroscience literature regarding how and where the brain encodes meaning. These perspectives are integrated into a theory in the form of a computational model trained from diverse data and prior knowledge, and capable of making experimentally testable predictions about the neural encodings and behavioral responses associated with tens of thousands of words, and hundreds of thousands of phrases and sentences.

This project potentially constitutes a significant scientific advance in understanding the relation between brain and mind, impacting a variety of scientific disciplines involved in the study of semantics, including linguistics, psychology, philosophy and cognitive science. A second impact comes from use of the methods and results to understand brain pathologies that involve language disturbances, such as aphasia, dyslexia, and autism. A third impact comes from the development of new statistical machine learning algorithms for analyzing and modeling cross-domain data sets to aid in scientific discovery. Finally, the emerging results and methods will have an educational impact through courses on Brain Imaging, Machine Learning, and Psychology taught by the Principal Investigators, and through a new course to be developed specifically on the topic of "Neural representations of meaning," with materials to be made available on the web.

DESCRIPTION (provided by applicant): This proposal seeks support to acquire a state-of-the-art Siemens Trio 3 Tesla MRI scanner system for the study of the brain systems underpinning various aspects of human cognitive and social functioning, as well as their typical and atypical development. Clinical and basic scientists from multiple disciplines and institutions stand poised to use the proposed scanner system to make field-leading contributions in the context of furthering their National Institutes of Health-funded, translational research programs. These research programs address critical questions in a number of areas including the cognitive neuroscience of autism, developmental cognitive neuroscience, aspects of high-level cognition, language processing and language pathology, MRI methods development, social neuroscience, and computational modeling of cognition and brain function. Scientists at Carnegie Mellon University and the University of Pittsburgh have been at the forefront of advances in MRI-based techniques particularly as they are applied to understanding the human mind and its development. While much progress has been made, fundamental questions about typical and atypical brain function remain to be addressed with the help of the requested instrument. PUBLIC HEALTH RELEVANCE: The work to be supported by the proposed MRI system focuses on the neural systems supporting cognitive and social functions, which are at the center of our emotional and intellectual well-being. In atypical neural systems, such as in autism, dyslexia, and stroke, the planned research will provide fundamental new insights that will help to improve human health and well-being. Many of the research programs that this scanner would support are in close concordance with current areas of emphasis for NIH, including autism, social neuroscience, and neurodevelopment.

DESCRIPTION (provided by applicant): This project proposes to discover the neural representation of some simple words and concepts. It uses newly developed machine learning techniques and dimension reduction methods applied to fMRI brain activation data that will be acquired in new experimental paradigms. This research approach has for the first time succeeded in identifying the content of individual human thoughts based on the pattern of brain activity. The initial published studies have demonstrated this capability in the case of concrete nouns and physical objects. This project proposes to expand the approach to a much larger set of different types of concepts and to examine the effect of the way the concept is presented (e.g. a written or spoken word, or a picture). The goal is to develop a comprehensive theory of how neural representations of meaning arise from the various brain systems that are used in interacting or considering the concept. An important secondary goal is to determine the degree of commonality of the neural representations across people. The studies propose to examine the neural dimensions of meaning in three domains: (a) physical objects; (b) human traits, emotions, and interpersonal interactions; and (c) small numerical quantities. This set of semantic domains is expected to provide sufficient breadth to reveal some of the principle neural bases of semantic representation. In contrast to the field of semantics (the study of the relation between words and their meanings), this project will help establish a new research area, neurosemantics, which is the study of the relation between words, thoughts, and their neural representations. The key assumption is that the underlying dimensions of meaning representation in the human brain are derived from basic neural systems. For example, one of the dimensions of representation of a physical object is how one physically interacts with or handles it. This dimension of representation is underpinned by a network of cortical areas that co-activate when one thinks about a physical object, and also when one actually handles the physical object. Other dimensions of neural representation similarly emerge when the concept is encountered. The studies will collectively identify the major dimensions of concept representation and relate them to networks of co-activating brain areas. The cumulative knowledge from the completed project will provide the framework of a theory of how brain systems map onto the representation of the meaning of concepts. The theory will be applicable to understanding and designing therapies for neurological conditions in which the meanings of concepts are distorted, such as Alzheimer's Disease, Pick's Disease, semantic dementia, and autism. The resulting theory will be foundational in relating the representation of meaning to brain function.

DESCRIPTION (provided by applicant): Over the past ten years a good deal has been learned from fMRI studies about the spatial patterns of neural activation used by the human brain to represent meanings of words and concepts. Much less is understood about the time evolution of this neural activity, including the temporally interrelated sub-processes the brain employs during the hundreds of milliseconds it takes to comprehend a single word, or the more complex processes it uses to construct and encode meaning of entire sentences as the words arrive one by one. We propose research to study, and to build computational models of, the detailed spatial and temporal neural activity observed during the comprehension of single words, phrases, sentences, and stories. This proposed research will specifically target the following questions: What information is encoded by neural activity where and when, and by which subprocesses in the brain, during the time it takes to comprehend a single word in isolation? What is the flow of information encoded when a newly sensed word first activates sensory cortex, then later results in neural activation encoding the word meaning? How does the brain integrate a newly encountered word in the context of earlier words in the sentence or phrase, to compose the meaning representation of the multi-word phrase or sentence? and How do semantic expectations and demands, together with syntactic sentence structure alter the processing of words, compared to processing the same words in isolation, or as an unstructured set such as {kick, Joe, ball}? To study these questions we will (1) devise novel experimental protocols to probe the flow of information encoded in neural signals during word and sentence processing, (2) collect new brain image data using both fMRI to achieve spatial resolution of a few millimeters, and MEG to achieve temporal resolution of a few milliseconds, (3) develop and apply novel machine learning approaches to build computational models that integrate and that predict this combined experimental data. Our goal is to develop an increasingly accurate computational model of how the brain comprehends words, phrases and sentences - a model that makes testable predictions about the neural activity observed in response to novel language stimuli. Intellectual Merit: This collaborative research brings together advanced machine learning algorithms with novel experimental protocols for MEG and fMRI brain imaging to advance our understanding of two fundamental open questions about the human brain: how does the brain represent meaning, and what neuro-cognitive processes construct that meaning piece-by-piece from perceived language stimuli? Broader Impacts: If successful, this research will impact a broad range of communities, including (1) cognitive neuroscience and computational linguistics, providing improved understanding of language processing in the brain, (2) machine learning, by driving the development of new methods for time series and latent variable analysis, integrating multiple data sets, and incorporating diverse background knowledge as priors, (3) clinical studies of brain pathologies, especially those related to language processing, and informing treatment strategies for developmental and acquired language disorders (4) education of graduates, undergraduates and the general public, through dissemination of technical articles, teaching materials, and news about our work in the public press.

Project Summary/Abstract The relationship between brain activation patterns in fMRI and the semantic features of concepts have been developed into bi-directional generative mappings in previous studies: they can extrapolate beyond the training stimuli and either predict neural activation patterns of a new word based on semantic features or predict the semantic features of a new word from the neural activation patterns it evokes. The current project will develop an analogous mapping between the semantic features and the EEG signals. Because fMRI is less portable and available than EEG, imparting this brain reading capability to EEG systems is desirable for clinical practice and in-home healthcare since it can potentially provide neurally-based diagnosis or direct brain communication for a variety of cognitive or psychiatric disorders. For example, altered social concept representations can serve as a thought marker for further screening of autism and patients with locked-in syndrome can communicate with caregivers using their interpreted EEG brain signals. Thus, this project has two aims. First, it will systematically find EEG features (e.g. Event-Related Potential, Event-Related Synchronization, etc.) that encode concept semantics and develop a mapping between these EEG features and semantic features. Second, it will bootstrap the semantic prediction accuracy of EEG by simultaneously acquired fMRI. Specifically, the mutual dependencies (e.g. mutual information, correlation) between the two recording modalities will be computed and used to relate the EEG features to their precise source locations. This can fulfill a long-awaited scientific promise of simultaneous EEG-fMRI: understanding the neural processing of concept semantics with both high spatial and temporal resolutions. Furthermore, this mutual dependency pattern will be constructed into a cross- participant bootstrapping mask to up-weight EEG features that are closely correlated with fMRI activation patterns. This mask will be applied to EEG acquired without fMRI to test the prediction accuracy on new words in new participants. In sum, this project will construct a systematic mapping between EEG features and concept semantics, and bootstrap the mapping by concurrent fMRI. These efforts will lead to the development of a portable and cost-effective concept interpreter, which can serve as a platform for screening psychiatric disorders by detecting altered concept representations or be engineered into future assistive devices for patients with communication disorders. Project Summary/Abstract

ABSTRACT: The assessment of suicidal risk is critical for treatment planning and monitoring of therapeutic progress for suicidal individuals. Current standard-of-care relies on patient self-report and clinician impression, which are not strongly predictive of imminent suicidal risk. This project advances a highly innovative approach to the assessment of suicidal risk, by using machine-learning detection of brain activation patterns that are neural signatures of individual concepts that have been altered in suicidal individuals. The overarching goal is to establish reliable neurocognitive markers of suicidal ideation (SI) and attempt (SA) in individual participants, and to assess these measures? ability to predict future ideation and attempts. In previous work, this approach was applied to the fMRI-based neurosemantic signature (NSS?s) during the thinking about each of 30 words related to either to suicide, negative concepts, or positive concepts in 17 SI young adults and 17 healthy controls (HCs). A machine learning classifier was able to discriminate between the SI and HCs with 91% accuracy, based on differential brain activation patterns in the L superior medial frontal cortex and anterior cingulate, areas known to be involved in self-referential thinking. Within the ideators, NSS?s also discriminated between those with a history of a SA from those without such a history with 94% accuracy. Moreover, using the classification algorithm derived from this sample, we were able to accurately classify a second sample of suicidal individuals with 87% accuracy. It was also possible to assess the emotions differentially manifested during the thinking about these words, and thus to differentiate SI from HC with 85% accuracy, and SI with and without SA with 88% accuracy. On the basis of these promising pilot findings, we propose to study 300 young adult SI (about half of whom will have made a SA), 100 never-suicidal psychiatric controls, and 100 HCs, use fMRI to assess NSS at intake and 3 months, and assess for suicidal ideation and behavior at intake, 3, 6, and 9 months thereafter. The goals are to determine if: (1) NSS?s are sensitive to changes in level of suicidal ideation when repeated at 3 months; and (2) whether NSS can predict trajectories of suicidal ideation and behavior upon prospective follow-up. We will also examine the relationship between NSS activation of circuits related to self-referential thinking and the death/suicide Implicit Association Test (IAT) that examines the extent to which a person associates suicide-related concepts with self. Finally, as a translational goal, we aim to develop and test a neurally based IAT that examines associations of suicidal concepts with self and with emotions as informed by NSS findings. This study, by shedding light on alterations in suicidal individuals? neural representation of suicide-relevant concepts could be extremely useful for: (1) identification of those with suicidal ideation who may not self-report their level of risk; (2) monitoring fluctuations in suicidal risk over time; (3) identification of emotional states associated with suicidal ideation; (4) guiding therapy to mitigate these alterations; and (5) the prediction of future suicidal ideation and behavior.

The premise of this project is that an understanding of how STEM concepts are organized in the brain would enable the enhancement of STEM concept learning and its assessment. Research has shown that although students vary in how accurately or completely they acquire knowledge related to a STEM concept, their neural filing systems are remarkably similar, in terms of which brain systems are the sites of particular aspects of concept knowledge. Understanding this neural organization common to everyone makes it possible to take that organization into account in the course of instruction. In effect, it makes it possible to “teach to the brain”. More specifically, it makes it possible to develop innovative cognitive training techniques based on modern machine-learning guided neuroscience to supplement traditional STEM learning. The goal is to formulate a detailed theory of how basic STEM concept knowledge in multiple STEM disciplines is neurally organized, how the underlying organization develops with learning, and how the organization is impacted by instructional and ability factors across STEM domains. Findings from this project would advance the understanding of the ontology of scientific concepts, theories of instructional design, and AI (machine-learning) guided instruction. Altogether these advances would facilitate more effective interventions towards expert-level knowledge. The design of this project will purposefully include both University and Community College students with a large range of STEM abilities in traditionally under-studied groups. <br/><br/>This project will assess the brain representations of STEM concepts (using several fMRI measures) in 4 different domains (physics, biology, chemistry, and mathematics) in students at multiple levels of expertise and assess the changes in those representations using machine-learning analysis of fMRI data under different types of instruction (class instruction, in-lab concept instruction, and expertise-focused training). The goal of the instruction will be to generate neural representations in novice learners that are similar to those of instructors or domain experts. This project builds on the investigators’ prior NSF funded work which demonstrated that fMRI can identify the underlying neural dimensions of physics concepts, and can predict and assess learning of these concepts (more so than traditional behavioral measurements). This research continues investigation of how neural data can usefully guide instruction by extending the approach to additional STEM domains and by guiding cognitive instruction with the accompanying neural assessment.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.