Deniz Erdogmus - US grants

Proposal Number: ECS-0524835

Proposal Title: Robust Information Filtering Techniques for Static and Dynamic State Estimation

PI Name: Ergodmus, Deniz

PI Institution: Oregon Health and Science University


Intellectual merit: This project addresses the well-established challenge of "state estimation," of estimating the current state of variables in a complex system which are not observed directly. Previous work in systems theory has developed well-perfected methods for systems whose variables are all continuous, or all discrete, under conditions where the dynamics of the system itself are already perfectly known. This team proposes a fundamental advance, by unifying recent breakthroughs addressing the challenge of what to do when the dynamical system is not perfectly known, in the general nonlinear case. The recent work to be drawn upon involves information-theoretic learning, sigma-point filtering, particle filtering concepts, and the use of recurrent neural networks and backpropagation through time (which offer major advantages as universal approximators of nonlinear dynamical systems).

Broader Benefits: Better state estimation will be important to all kinds of challenges in managing complex systems more effectively, whether by neural networks or other components. The testbed to be used here- the localization of elderly patients in a prototype advanced health care clinic - was chosen both for its value as a challenge to the basic research and for its promise as a starting point for large real-world benefits.
The educational benefits included cross-disciplinary education (highly credible, given the team and the project) plus more standard sorts of benefits to education.

NSF-ECS Proposal 0622239

Nonparametric Nonlinear Adaptive Detection and Estimation

Objectives and approaches: The objective of this research is to create a unifying nonlinear information processing framework to handle increasingly complex modeling and data analysis problems encountered in science and engineering. The proposed framework exploits and unifies three concepts: reproducing kernel Hilbert spaces, nonparametric kernel density estimation, and information theoretic optimality measures. Developed techniques will be illustrated in a brain interface, in which high dimensional spatiotemporal electroencelaphogram activity will be translated to intended commands for a robot. This interface will be designed to facilitate fusion with myoelectrically controlled neural prostheses.

Intellectual merit: Conventional techniques relying on nonlinear programming of semiparametric models with low-order statistical criteria, which is prone to various difficulties including model complexity selection and existence of suboptimal solutions, cannot cope with the challenges of increasingly complex engineering problems. Proposed work facilitates the adaptation of nonlinear models exploiting smooth generalized linear models and principled information processing in a novel unified framework. This will advance the state-of-the-art in nonlinear adaptive signal processing and machine learning. The brain interface testbed will facilitate future collaboration between two currently disconnected neural prostheses communities.

Broader impacts: The theoretical framework will impact signal processing and machine learning, thus will have immediate influence on fields that rely heavily on statistical modeling, data analysis, and processing techniques. Specifically, contributions to the emerging neural engineering field will impact the design of future neural prostheses, as well as fundamental brain and cognition research. The project will help educate graduate engineers with mathematical rigor and an interdisciplinary focus.

This is a project to develop new methods for scientifically studying and assessing human cognitive function. It will employ sophisticated statistical multimodal data analysis techniques that will fuse contextual, behavioral, and neural information simultaneously obtained from human beings in the process of completing complex batteries of cognitive tasks. The tasks will be presented in the form of customized computer games that are designed to exhibit the crucial aspects of established cognitive assessment tests and at the same time provide a motivating and engaging environment for the subject's interactions with the game and computer agents. The tasks will involve exploiting our existing capabilities of monitoring and controlling certain enjoyable and challenging computer games that involve various combinations of cognitive tasks ranging from working memory and attention to executive functions. Multimodal information fusion will be accomplished by utilizing Bayesian inference techniques and information theoretic data analysis and dimensionality reduction methods.

The work to be carried out under this grant aims to develop sophisticated pattern analysis techniques for the purpose of analyzing the fine-grain behaviors of elderly when they are engaged in complex cognitive tasks in the form of computer games. Expected significant scientific findings from the proposed research are two-fold: (1) improved statistical signal processing and pattern recognition algorithms for EEG processing, (2) an enhanced understanding of the interplay of multiple cognitive processes and their neural signatures in EEG during the execution of complex tasks.

The approach is innovative in terms of three aspects: (1) an advanced adaptive interaction protocol that modifies the task parameters to maintain maximal sensitivity to cognitive state changes will be employed, (2) novel information theoretic techniques will be developed and utilized for the extraction of maximally discriminative features from EEG measurements for cognitive state estimation and neural activity visualization, (3) the developed closed-loop system will be utilized to study the human-agent interaction in complex cognitive tasks resulting in mathematical models of micro-behavior in realistic evolving environments as opposed to traditional stationary repetitive experimental paradigms.

The successful completion of the work will open the way to further collaborative activities in brain interface design, closed-loop collaborative augmented cognition human-agent interfaces for improved performance, and early diagnosis of cognitive decline in elderly. An interdisciplinary research environment will engage the participating graduate students in a multidisciplinary educational setting and will help them develop skills to perform collaborative interdisciplinary research.

In this project the PI will address the challenge of empowering people with severe motor and speech impairments (SMSI) to socialize through written and spoken language, by increasing communication rate through a novel and intuitive computer interface. Available augmented communication technologies for the SMSI population typically yield speeds on the order of just one word per minute (based on clinical experience). The PI's objective is to develop an EEG-based brain interface technology based on an intuitive icon-based language generation framework, RSVP iconCHAT, which will achieve increased communication rates for the target population. This technology will exhibit three essential features: rapid serial visual presentation (RSVP) of icons that represent words; a large-vocabulary natural language model with the capability for accurate predictions of intended text in order to control the upcoming sequence of icons to be shown to the subject for confirmation in the RSVP paradigm; and an intent detection mechanism that fuses information from multichannel electroencephalography (EEG) and the generative probabilistic language model. Advanced statistical signal processing, machine learning, and natural language modeling techniques will be employed to achieve communication rates over an order of magnitude higher than the current state-of-the-art. The project will also contribute novel techniques and algorithms for synchronous brain interface design, particularly single-trial ERP detection. Both the brain interface and language model components will learn from previous interactions with the user and exhibit robust cooperative learning behavior in order to maximize language throughput. A Bayesian and information theoretic foundation will support adaptability. The PI notes that his approach is innovative along three dimensions: an intuitive icon-based language representation combined with context-dependent language models will be employed for message construction; a noninvasive brain computer interface that is user-adaptive will be developed and employed to interface with the icon-based platform; and methods for probabilistic information fusion between the brain activity measured by the BCI and the predictive language model will be developed.

Broader Impacts: There exists a significant SMSI population due to various reasons such as cerebral palsy (CP), neuromuscular disease (Amyotrophic Lateral Sclerosis, ALS), and severe spinal cord injury leading to locked-in syndrome (LIS). These communities rely on inefficient modes of communication that limit the user's ability to generate acceptable communication rates. Successful achievement of this project's goals will not only provide the target population with an improved face-to-face communication experience with their able-bodied communication partners, but will also enable control of their environment and access to information. In addition, the work will contribute to information fusion from different modalities, optimal data dimensionality reduction, single-trial ERP detection, and human computer communication through a novel interface. Data collected in experiments will be made available to other researchers in order to accelerate verification of outcomes and dissemination of results.

The goal of this project is to develop methods that will permit researchers to remotely and automatically monitor behavior of primates and other highly social animals. The PIs will collect behavioral data from cameras and microphones. They will then develop statistical models and computational algorithms to track the individuals in the group and to recognize facial expressions and vocalizations. Patterns in movements, expressions, and vocalizations will be used to develop behavior-identifying algorithms that will recognize different behaviors such as aggression, submission, grooming, eating and sleeping. The project is a collaboration between computer scientists and primatologists. A key element of this project is the observation that complex social interactions can often be regarded as being composed of sequences of elementary behaviors which occur frequently and consist of relatively simple and distinct gestures. Thus, the task of modeling complex social interactions can be broken down into two regimes ? elementary behaviors spanning short duration, and their stochastic sequences spanning relatively longer time duration.

Apart from advancing computational science, the new methods for recording behavior unobtrusively and analyzing them at a high data rate are likely to be of interest to behavioral ecologists, socio-biologists and neuroscientists in studies of primates and other highly social animals. With these new tools, scientists can study and understand behavior, for example, in the context of planning conservation efforts for threatened species, building accurate animal models for health research, and supporting animal husbandry decisions in zoos. The project will provide an extensive, annotated data repository and associated algorithms and will also fund graduate students who will gain hands-on training in all aspects of the project.

Hospital clinical laboratory tests are a major source of medical information used to diagnose, treat, and monitor patients. Such test errors lead to delays, additional clinical evaluation, additional expense, and sometimes to erroneous treatments that increase risk to patients. One recent study suggests that errors in measured total blood calcium concentration due to instrument mis-calibration alone cost from $60M to $199M annually in the US. However, the vast majority of clinical laboratory errors do not originate in instrument mis-calibration. Clinical laboratory errors affect about 0.5% of samples collected. Of those, approximately 75% of clinical laboratory test errors originate during sample collection, transport, and storage before samples reach the analysis instruments i.e., the pre-analytic phase. However the quality control measures standard in hospital clinical test labs only monitor instrument calibration and are therefore completely blind to sample faults introduced in the pre-analytic phase, where most errors originate. Data derived from patient samples, rather than instrumentation calibration checks, holds the key to detect faults introduced in the pre-analytic phase. Current methods are either so insensitive to errors that they do not detect sample faults reliably, or they routinely flag normal samples as being faulty.

This project brings together an interdisciplinary team of researchers from Oregon Health and Science University and Northeastern University with expertise in machine learning, signal processing, and laboratory medicine to develop and apply statistical machine learning technology to reliably detect errors in hospital clinical laboratory tests, using data derived from patient samples. The primary obstacle to developing reliable statistical detectors for lab errors is the cost of labeling samples combined with the low error rate. Developing and evaluating any automated error-detection algorithm requires a sufficient number of samples, both faulty and non-faulty. Determining which tests are faulty requires review of the tests and other patient data (e.g. charts) by a clinical lab expert - a time-consuming and economically unfeasible prospect given the low fault rate. The project addresses this challenge through active learning paradigms used to select, with emphasis on rare classes, subsets of the data for labeling by human experts. The project focuses on chronic kidney disease because of its medical importance and large data repository at Oregon Health and Science University. This research will provide algorithms for clinical lab error detection that will extend to tests used in other disease entities (for example diabetes and heart failure).

Ultimately, the error-detection algorithms developed from this research will make their way into clinical laboratory information systems and further into commercialization and thus deployment on a scale significant enough to have widespread positive impact on laboratory costs patient risk. The project provides cross-disciplinary training in statistical pattern recognition and clinical laboratory science for graduate and undergraduate students. Additional information about the project can be found at: http://www.bme.ogi.edu/~tleen/LabErrorDetect/.

This project develops a framework for design automation of cyber-physical systems to augment human interaction with complex systems that integrate across computational and physical environments. As a design driver, the project develops a Body/Brain Computer Interface (BBCI) for the population of functionally locked-in individuals, who are unable to interact with the physical world through movement and speech. The BBCI will enable communication with other humans through expressive language generation and interaction with the environment through robotic manipulators.

Utilizing advances in system-level design, this project develops a holistic framework for design and implementation of heterogeneous human-in-the-loop cyber-physical systems composed of physically distributed, networked components. It will advance BBCI technology by incorporating context aware inference and learning of task-specific human intent estimation in applications involving semi-autonomous robotic actuators and an efficient wireless communication framework.

The results of this project are expected to significantly speed up the design of complex cyber-physical systems. By accelerating the path from idea to prototype, this work shortens the time frame of and cost of development for assistive technology to improve the quality-of-life for functionally locked-in individuals. This project establishes an open prototyping platform and a design framework for rapid exploration of other novel human-in-the-loop applications. The open platform will foster undergraduate involvement in cyber-physical systems research, building confidence and expertise. In addition, new activities at the Museum of Science in Boston will engage visitors to experiment with systematic design principles in context of a brain computer interface application, while offering learning opportunities about basic brain functions.

Proteins are molecular machines that contribute to virtually every activity of every biological system. Regulation of their activity is a principal goal of drug and protein design. But we do not yet know enough about how proteins function to efficiently design molecules that modulate their activities. We know that to function they need to change their shape - alter their conformation - but visualizing these changes in shape is a substantial experimental challenge. The approach we will take here is to attempt to characterize the set of all shapes a protein can take on - its full conformational ensemble. This will provide a critical link between structure, dynamics and function. The problem is that in a drop of solution, there is a multitude of proteins, each of which may have a different conformation. We will alter the relative abundance of each conformation under many different experimental conditions and collect wide-angle x-ray solution scattering (WAXS) data from the protein under each of those conditions. Using advanced signal processing techniques, we will then extract from these data the scattering due to each conformation individually. This will provide direct structural information on the conformations of functional intermediates that never occur in solution in the absence of other conformations. The result will be a map of the conformational changes that occur during protein action, providing direct experimental evidence for understanding the way proteins use conformational changes to carry out their functions.

The project will make extensive use of state-of-the-art signal processing techniques that are well known in engineering but used sparingly in biophysics. This exemplifies a rapidly accelerating trend to data-rich scientific inquiry powered by increasing use of automatic data capture. Science is becoming ever more dependent on sophisticated methods of signal and data analysis. Training of scientists in advanced data processing techniques will be an essential part of this transition. In this project, success will depend on cross-disciplinary training of both scientists and engineers; training that starts with the PI's - who are already learning from one another - continues with graduate students involved in the research, and extends to graduate students and undergraduates who will benefit from the cross-disciplinary courses that we will create. Underrepresented groups will be introduced to this approach through research experiences for undergraduates and K12 teachers and outreach to K12 students - activities that will constitute an integral part of this project and be designed to encourage and prepare students for careers in science or science teaching.

The PI's ultimate research goal is to empower people with severe speech and physical impairments so they can live their lives to the fullest extent independently and productively. To this end, he will in this project exploit and advance emerging brain computer interface (BCI) technology by rigorously developing macro-level dynamic models for the visual evoked potentials (VEP) in the brain measured by electroencephalography (EEG) in the context of BCI design. The models will enable a communication channel interpretation of the BCI and will allow analysis and design breakthroughs stemming from the application of information theory and digital communication concepts. Cortical dynamics and background processes will be modeled using a probabilistic dynamic framework at a spatiotemporal scale appropriate for BCI analysis and design. Model-based performance limits on bandwidth and calibration accuracy will then be determined, in order to develop better information coding techniques for optimal communication bandwidth (speed) utilization and better subject training and model calibration procedures for best accuracy return on investment of effort. Prototype real-time applications that operate at optimal or near-optimal performance levels utilizing the developed theoretical advancements for communication and control will be implemented, to enable access by and support independence for the target user groups.

Project outcomes will disrupt the trend of black-box BCI design by building dynamic system models for stimulus-to-EEG systems encountered in BCI applications, and treating them as stochastic communication channels in order to characterize signals accordingly and to employ information theoretic approaches to analysis and design. This novel theoretical framework will enable model-based quantitative characterization of BCI performance limits and will allow the design of optimal or near-optimal coding/decoding strategies as well as improved calibration procedures that will have immediate impact on increasing bandwidth and intent detection accuracy, as well as calibration duration reduction in BCI systems - primary barriers between laboratory prototypes and real-world-worthy BCI products.

Broader Impacts: If successful this project will advance BCI technology to the next level, thereby revolutionizing human computer interaction and empowering persons with physical disabilities by enabling seamless control of computers and devices. The project will afford, through collaboration with the Center for Subsurface Sensing and Imaging Systems (CenSSIS) at Northeastern University as well as colleagues across departments and institutions, opportunities to both undergraduate engineering and non-engineering majors for enhanced learning and collaboration skills by immersing them in interdisciplinary cutting-edge research and design projects with societal impact. The PI will engage high school students and teachers in the research through his institution's Center for STEM Education. And he will inform the broader public of ongoing technological advances in the BCI field and raise disability awareness through collaboration with the Cahners ComputerPlace at the Boston Museum of Science.

Part 1: Upper-limb motor impairments arise from a wide range of clinical conditions including amputations, spinal cord injury, or stroke. Addressing lost hand function, therefore, is a major focus of rehabilitation interventions; and research in robotic hands and hand exoskeletons aimed at restoring fine motor control functions gained significant speed recently. Integration of these robots with neural control mechanisms is also an ongoing research direction. We will develop prosthetic and wearable hands controlled via nested control that seamlessly blends neural control based on human brain activity and dynamic control based on sensors on robots. These Hand Augmentation using Nested Decision (HAND) systems will also provide rudimentary tactile feedback to the user. The HAND design framework will contribute to the assistive and augmentative robotics field. The resulting technology will improve the quality of life for individuals with lost limb function. The project will help train engineers skilled in addressing multidisciplinary challenges. Through outreach activities, STEM careers will be promoted at the K-12 level, individuals from underrepresented groups in engineering will be recruited to engage in this research project, which will contribute to the diversity of the STEM workforce.

Part 2: The team previously introduced the concept of human-in-the-loop cyber-physical systems (HILCPS). Using the HILCPS hardware-software co-design and automatic synthesis infrastructure, we will develop prosthetic and wearable HAND systems that are robust to uncertainty in human intent inference from physiological signals. One challenge arises from the fact that the human and the cyber system jointly operate on the same physical element. Synthesis of networked real-time applications from algorithm design environments poses a framework challenge. These will be addressed by a tightly coupled optimal nested control strategy that relies on EEG-EMG-context fusion for human intent inference. Custom distributed embedded computational and robotic platforms will be built and iteratively refined. This work will enhance the HILCPS design framework, while simultaneously making novel contributions to body/brain interface technology and assistive/augmentative robot technology. Specifically we will (1) develop a theoretical EEG-EMG-context fusion framework for agile HILCPS application domains; (2) develop theory for and design novel control theoretic solutions to handle uncertainty, blend motion/force planning with high-level human intent and ambient intelligence to robustly execute daily manipulation activities; (3) further develop and refine the HILCPS domain-specific design framework to enable rapid deployment of HILCPS algorithms onto distributed embedded systems, empowering a new class of real-time algorithms that achieve distributed embedded sensing, analysis, and decision making; (4) develop new paradigms to replace, retrain or augment hand function via the prosthetic/wearable HAND by optimizing performance on a subject-by-subject basis.

The broader impact/commercial potential of this I-Corps project is to help millions of individuals with chronic or acute disabilities leading to loss of communication and computer control abilities. The proposed assistive context aware interface (ACAI) will allow these individuals to regain the ability communicate with caregivers and families, and to control their environments, which will lead to increase in quality of life and in some cases improvement in received healthcare. Potential customers include close to 4 million people worldwide, with conditions such as spinal cord injuries, strokes, multiple sclerosis, amyotrophic lateral sclerosis, and traumatic brain injury. This project will pursue the commercialization of ACAI as a stand-alone computer interface with which individuals can use existing assistive technology, including augmentative and alternative communication solutions that targets individuals as customers, and its commercialization as part of a complete intensive care unit delirium assessment system that will be offered to hospitals for improved patient care.

This I-Corps project will offer an infrastructure that supports rich contextual information exchange between physiological and behavioral sensors that capture human intent for the control of computer applications. The assistive context aware interface (ACAI) framework is based on Bayesian inference and information theoretic coding principles that ensure mathematical rigor in design in offering almost optimal speed-accuracy performance to the user. The framework is based on the human-in-the-loop cyber-physical systems design principles, which ensures a user-centric, modular and scalable design for assistive computer access using all physiological and behavioral signals that can be exploited by the users in their clinical conditions. ACAI unifies body and brain physiological signal processing in human intent inference. Convenient user customization will allow users to exploit multiple input modalities to control assistive computer applications, and promises an adaptive solution that can cater to their changing needs during treatment or disease progression.

Retinopathy of prematurity (ROP) is a leading cause of childhood visual loss worldwide, and the social burdens of infancy-acquired blindness are enormous. Early diagnosis is critically important for successful treatment, and can prevent most cases of blindness. However, lack of access to expert medical diagnosis and care, especially in rural areas, remains a growing healthcare challenge. In addition, clinical expertise in ROP is lacking, and medical professionals are struggling to meet the increasing need for ROP care. As point-of-care technologies for diagnosis and intervention are rapidly expanding, the potential ability to assess ROP severity from any location with an internet connection and a camera, even without immediate ophthalmologic consultation available, could significantly improve delivery of ROP care by identifying infants who are in most urgent need for referral and treatment. This would dramatically reduce the incidence of blindness without a proportionate increase in the need for human resources, which take many years to develop.

This project develops a prototype assistive integrative support tool for ROP, featuring a modular design comprising: (a) image analysis, (b) information fusion of clinical, imaging, and diagnostic data, and (c) generative probabilistic and regression models with associated computationally efficient machine learning algorithms. The outcomes of the project include disease severity metrics and diagnostic estimates obtained through clinical evidence classifiers trained jointly over expert-generated labels. These labels consist of discrete diagnostic labels, as well as comparison outcomes of relative severity between pairs of images. Random process models for vessel tortuosity and diameter distributions over the retina, as well as patch-based vessel-free image analysis through the use of convolutional neural networks on the entire image, enhance and augment feature extraction. Moreover, incorporating severity comparison outcomes through novel hard and soft constraint methods force inferred severity to agree with ordinal information provided by experts and address inherent uncertainty in expert ground-truth labels. The above severity inference methods are evaluated and fine-tuned over a broad array of generative models, both through retrospective analysis, including cross-validation, longitudinal tests, and tests across multiple sites, as well as through prospective analysis, evaluating its real-world clinical impact.

Spatial presence, in Virtual Reality (VR) terminology, refers to the perception (or illusion) of being physically present in a simulated environment. VR strives to create interactive environments that provide experiences of spatial presence through accurate delivery and perception of multimodal sensory stimuli. Research in VR spans fields ranging from neuroscience and medicine to gaming. While the computing and gaming industries have generated tremendous advances in hardware and software for graphics processing and 3D display technologies, VR systems still lack capabilities for providing users with haptic feedback (a sense of touch), which is crucial for generating truly immersive, real-world experiences. It is known that an increase in the feeling of spatial presence manifests itself in the form of increased brain activity. This research aims to achieve the control of haptic sensory stimulation adaptively, based on the changes in brain activity associated with perceptual responses elicited by sensory stimulation in VR environments. Project outcomes will include novel scientific discoveries and engineering enhancements that will make significant contributions to other areas of interest, such as prosthetic limbs, augmented reality, and telepresence applications. The project will help train a new generation of engineers skilled in addressing multidisciplinary challenges, while through outreach activities STEM careers will be promoted at the K-12 level.

The research objective is to identify and analyze brain activity associated with the increased feeling of haptic spatial presence elicited by electro-tactile stimulation and measured through EEG, and to investigate closed-loop techniques to control electro-tactile stimulation for enhanced haptic presence in VR environments. Specifically, the project will: (1) develop an electrical haptic stimulation framework; (2) design analysis techniques to identify markers of haptic inputs in EEG; (3) establish control policies for adaptive electrical stimulation; and (4) evaluate and refine EEG-guided adaptive stimuli control framework in VR environments. In particular, the proposition to actuate haptic feedback through electrical stimulation is novel, while formulating design principles for model-based optimal EEG-guided closed-loop haptic feedback for immersive spatial presence is transformative. Additional innovative propositions to advance adaptive control under uncertainty and psychophysical investigations are unique; these present a potentially game-changing opportunity for VR system development and perhaps for general human-computer interaction.

This project addresses a question that has vexed scientists for more than a century: how does the motor cortex (the part of the brain where nerve impulses initiate voluntary muscular activity) represent and coordinate multiple muscles in order to produce a vast range of movements? To answer this question, this project will harness the unique strengths of non-invasive, navigated, transcranial magnetic stimulation (TMS) mapping to establish causal links between brain physiology and behavior. TMS is achieved by placing a coil of wires near the scalp, which when activated with an electrical current will create a magnetic field across the scalp and skull to stimulate the brain. TMS is the only non-invasive method available to stimulate the brain like invasive stimulation. However, to use TMS-based motor mapping to understand multi-muscle physiology and control, innovations in three areas are critically needed: 1) drastically improving the efficiency, efficacy and reliability of the TMS-based motor cortex mapping processes, 2) characterizing and validating TMS-based mapping as a probe for understanding the relationship between multi-muscle activation and voluntary movement, and 3) applying a neural network computational method to improve understanding of motor control and organization. Enhanced understanding of motor cortex physiology through TMS mapping of motor representations has the potential to better map the brain in applications such as surgical removal of tumors, assessing brain injury due to concussions or stroke, and identifying cortical networks needed for successful brain-machine interactions for controlling prostheses. Students involved with this project will be trained to address multidisciplinary challenges at the intersection of neuroscience, non-invasive brain stimulation, software design, control theory, machine-learning, statistical signal processing, data dimensionality reduction and visualization. Partnership with Boston-based leaders in the technology industry will provide state-of-the-art training to undergraduate, graduate, and post-graduate trainees. Through cooperative educational programming at Northeastern University and internships with Mass General Hospital, STEM-based learning opportunities will be provided for middle- and high-school students, inspiring a diverse body of students to pursue STEM careers. To promote STEM careers and demonstrate impact, the team will reach out to local venues that promote public awareness and appreciation of science, such as science fairs and the Boston Museum of Science.

The goal of this collaborative project is to develop a deeper mechanistic understanding of the role of the motor cortex (M1) in controlling single muscles and synergies in producing complex movements. This will be accomplished by developing several innovations in the use of non-invasive transcranial magnetic stimulation (TMS) to map the spatial distribution of synergies and single muscles. Transformative computational advances will be used to extract more accurate information about brain interaction with other physiological systems outside the motor domain and increase the rigor of analysis and data visualization to enhance interpretability, and repeatability. An enhanced understanding of corticomotor organization of complex movement will pave the way to studying motor system development across the lifespan, the basis of human performance enhancement, and the basis and characterization of neuromotor diseases. The research plan is organized under 3 aims. AIM 1 is to accelerate acquisition of TMS-based maps by developing an active learning process based on a Gaussian Process Model (GPM) of Muscle Evoked Potentials (MEPs) as a function of 2D spatial coordinates on the scalp. The developed Active-GMP learning algorithm is expected to speed up the mapping process by diverting time spent on loci with null data to loci where the model needs more samples to improve certainty. The efficacy and the accuracy of the new algorithm will be compared to three existing alternatives. AIM 2 is to test the behavioral relevance of synergies derived from human multi-muscle TMS mapping, i.e., to biologically validate the technical methods developed in Aim 1. Specifically, TMS and Voluntary (VOL) EMG data will be collected from 16 hand-arm muscles in healthy participants while subjects mimic hand postures for static letters and numbers of the American Sign Language alphabet. Non-negative matrix factorization-extracted synergies from VOL data and TMS data will be compared to determine if the TMS-elicited synergies match those utilized during movement production and if the adaptive Active-GMP and user-guided approaches more closely match synergies derived from VOL data compared to other approaches. AIM 3 is to develop generative and inverse topographic imaging models that allow forward modeling of M1 control and reverse mapping of M1 organization, respectively, of muscles and synergies. Hybrid models combining subject-specific FE modeling of TMS-induced cortical electric fields with neural network models trained to predict evoked muscle responses will be used to answer key questions: Q1) Are synergies dominant features of motor control? Q2) Do direct M1 motorneuron projections augment a synergy model of control? and Q3) Are muscles and synergies discretely organized in M1?

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Project Title: D3SC: Mining for Mechanistic Information to Predict Protein Function

Proteins perform a variety of essential functions in a cell, including catalyzing chemical reactions as enzymes. With this award, the Chemistry of Life Processes Program in the Chemistry Division is funding Dr. Mary Jo Ondrechen, Dr. Penny Beuning and Dr. Deniz Erdogomus at Northeastern University to develop new ways to predict the function of a protein from its three-dimensional structure. This computational problem is a major challenge in genomics - the study of DNA sequences and their protein products. Research in genomics is opening the door to tremendous current and future innovations to benefit society, in areas as diverse as food production, energy, the economy, the environment, and health. In this project, chemical properties are computed and coupled with machine learning algorithms to identify the specific biochemical roles for the active amino acids in a protein structure, which then leads to the prediction of the protein's function. These predictions of function are tested experimentally by direct biochemical assays and by ligand binding studies, for selected cases. Doctoral students and undergraduate research interns, including those from minority groups that are underrepresented in STEM fields, are being trained through this project to become highly qualified scientists in the areas of computational chemistry, informatics, machine learning, and biochemistry. These skills are vital to the regional high-tech economy of New England and to United States competitiveness in the global economy.

The computational prediction of biochemical functional roles of individual amino acids in a protein structure is entirely new. The predictive power of properties obtained from computational chemistry are being enhanced by machine learning approaches, including Support Vector Machines (SVM) and Graph Convolutional Neural Networks (GCNN). Improved, experimentally tested methods for the prediction of protein function contribute significantly to the interpretation of the massive quantities of data from genome sequencing and Structural Genomics (SG) initiatives. A significant feature of this project is that it incorporates computed chemical properties of the amino acids in a protein structure into more conventional informatics methods to predict function, whereas most current methods are purely informatics-based approaches. This project is unique in that it employs computed chemical reactivity and electrostatic features on the atomic scale in the protein function prediction problem to obtain residue-specific mechanistic information. With the capability to match functional types across different structural folds, i.e. cases with neither sequence nor 3D structure similarity, the ability to assign biochemical function reliably is substantially increased for SG proteins of unknown or uncertain function. This work also leads to better understanding of how enzymes work and of how specific amino acid residues achieve their catalytic power.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The research objective of this project is to improve the fluidity of human-robot interactions wherein robots and humans can seamlessly pass objects between one another. Object handover is critical to everyday interactions, whether in ordinary environments or in high-stakes circumstances such as operating rooms. Seemingly effortless object handover results from successful inference and anticipation of shared intentions and actions. Manual object transfer between humans and robots will become increasingly important as robots become more common in the workplace and at home. The project team will perform human subject experiments investigating human-human and human-robot interactions within the context of object handover tasks to identify characteristics of dyadic coordination that allow people to understand their collaborator's intentions, to anticipate their actions, and to coordinate movements leading to task success. The team will use that new knowledge to develop robots that people can collaborate with on physical tasks as readily as they do other humans. Broader Impacts of the project include training opportunities for high school, undergraduate, and graduate students, with efforts to increase participation of underrepresented groups.

This project explores human and robotic perception, behavior, and intent inference in a bidirectional and integrated manner within the context of object handover. Three specific aims are planned. The first uses motion capture, eye tracking, and electroencephalography data to build models of human intent and action during object handover. A novelty of the models is that they are sensitive to an individual's role (giver vs. receiver; leader vs. follower), to the presence or absence of communicative gaze, and to the degree of predictability of certain aspects of handover, such as grasp type, locus of handover, gaze conditions, and dyadic role. The second aim will develop a real-time intent inference engine that uses recursive Bayesian state estimation to obtain a probabilistic assessment of human intent as the handover action evolves in time. The output of this model will inform the robot's trajectory planner, thereby enabling it to make short time horizon predictions of human actions and to adjust robot motion plans accordingly in real-time. The third aim will examine how specific choices in the high-level planning and low-level control of robot motion impact human inference of robotic intent and action during object handover. If successful, this project will advance understanding of robot manipulation during human-robot handover and yield algorithms for achieving advanced autonomy during human collaboration with humanoid robots.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The research objective of this proposal, Multimodal Signal Analysis and Data Fusion for Post-traumatic Epilepsy Prediction, with Pl Dominique Duncan from the University of Southern California, is to predict the onset of epileptic seizures following traumatic brain injury (TBI), using innovative analytic tools from machine learning and applied mathematics to identify features of epileptiform activity, from a multimodal dataset collected from both an animal model and human patients. The proposed research will accelerate the discovery of salient and robust features of epileptogenesis following TBI from a rich dataset, collected from the Epilepsy Bioinformatics Study for Antiepileptogenic Therapy (EpiBioS4Rx), as it is being acquired by investigating state-of-the-art models, methods, and algorithms from contemporary machine learning theory. This secondary use of data to support automated discovery of reliable knowledge from aggregated records of animal model and human patient data will lead to innovative models to predict post-traumatic epilepsy (PTE). This machine learning based investigation of a rich dataset complements ongoing data acquisition and classical biostatistics-based analyses ongoing in the study and can lead to rigorous outcomes for the development of antiepileptogenic therapies, which can prevent this disease. Identifying salient features in time series and images to help design a predictor of PTE using data from two species and multiple individuals with heterogeneous TBI conditions presents significant theoretical challenges that need to be tackled. In this project, it is proposed to adopt transfer learning and domain adaptation perspectives to accomplish these goals in multimodal biomedical datasets across two populations. Specifically, techniques emerging from d,eep learning literature will be exploited to augment data, share parameters across model components to reduce the number of parameters that need to be optimized, and use state-of-the-art architectures to develop models for feature extraction. These will be compared against established pipelines of hand-crafted feature extraction in rigorous cross-validation analyses. Developed techniques for transfer learning will be able to extract features that generalize across animal and human data. Moreover, these theoretical techniques with associated models and optimization methods will be applicable to other multi-species transfer learning challenges that may arise in the context of health and medicine. Multimodal feature extraction and discriminative model learning for disease onset prediction using novel classifiers also offer insights into biomarker discovery using advanced machine learning techniques through joint multimodal data analysis.

Understanding the brain’s role in behaviors such as movement, cognition and emotion is paramount to progress in science and engineering, and to advancing improvements in health and wellness. Invasive approaches in animals cannot be readily adapted to humans, creating a technological barrier to causal study of the brain in awake behaving humans. One promising approach in humans, transcranial magnetic stimulation, uses magnetic pulses to noninvasively and safely modulate brain activity. However, stimulators modulate brain cells (neurons) indiscriminately, which prevents studying how distinct neurons drive behavior. This award will facilitate the acquisition of a cutting-edge stimulator that allows scientists to modulate specific neuron populations in the brain. The system includes an integrated positioning robot for precise localization and recording devices that read physiological signals from the brain or muscles to objectively quantify the effects on different neural populations and behavior. This instrumentation will enable discoveries that will catalyze new research in the study of brain and behavior. Crucially, the instrumentation paired with the proposed education plan will create unique training opportunities for students in STEM and health science, lowering the barrier of entry for underrepresented students, including persons of color and women. The project leverages Northeastern University’s experiential education model and various diversity/inclusion initiatives to support research by diverse (under)graduate and K-12 students and teachers.

The project proposes the acquisition of a controllable-pulse transcranial magnetic stimulator capable of differentially modulating specific neural populations in the human brain, with integrated robotic positioning and electroencephalography and electromyography recording. This instrumentation will be the only such system in the Northeastern US. As part of the Northeastern University Non-Invasive Brain Stimulation Center, it will enable unprecedented basic science research into human neurophysiology and brain-behavior relationships, and significant advances in fundamental engineering research in stimulator development, automated robotic positioning, stimulation-induced artifact removal in physiological recordings, closed-loop stimulation, and artificial intelligence / machine learning algorithms. Allowing researchers to control stimulus waveforms and to differentially activate distinct neural populations will enable a scientific scope of work that will transcend multiple disciplines including motor and affective neuroscience, cognition, memory, development, aging, and biophysical modeling of brain physiology. The proposed diversity and education plan enabled by this instrument will lower barriers for underrepresented minority students to engage in cutting-edge experiential STEM education. The project’s impact will be profound on science and technology innovation, as well as in training a new, diverse, interdisciplinary workforce to drive this field forward.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.