Tzyy-Ping Jung - US grants

The project proposes to create a distributed, multi-user social computing environment that will develop the capabilities of human Electroencephalography (EEG) to analyze users engagement with digitally based experiences. For this project, users will wear non-invasive, EEG headsets while navigating a shared virtual world. Beginning with a handful of EEG systems, the team will scale up over the course of the project to gather signals from dozens of users, providing a basis for larger scale studies. By comparing the EEG signals with each participants activities in virtual world, and with the brain activity and the activities of other users, a model of human brain activity will be developed for different types of behavior profiles and subjective states. This will allow significant improvement for the development of neural markers of human perceptual, cognitive and affective states, the parsing of EEG signals, the applicability of EEG interfaces to new types of experiences, all of which can enhance distance learning, collaborative distributed work, improved mobile computing interfaces and health care applications. The project will advance the capabilities for determining an individuals cognitive state by the creation of new computing methods utilizing comparative EEG analysis and data analysis of event states in a digital simulation. Bringing methods of large scale data analysis to articulate patterns across many users in the situated milieu of the online virtual world will create a new method to utilize EEG analysis to infer human subjective experience. The necessity of conducting this analysis in real-time, with data gathered from distributed, wireless EEG instruments will provide the impetus for utilizing accelerated hybrid multi-core techniques to bear on this domain.

The results from the project will be applicable for a variety of digital environments including computer aided learning and training, digitally mediated collaborative work environments, visualization of complex data sets, and digitally based entertainment experiences such as virtual worlds and computer games. The project will improve the functionality and outcomes of these digital media environments, better adjusting them to the cognitive states of the users. The PIs will train and employ a diverse body of participants to be involved in these activities. As part of an internship program at the Preuss School, a nationally recognized middle and high school on the campus of UCSD, UCSD faculty and staff mentor high school seniors to provide these students valuable experiences in a research laboratory. In addition, UMBC operates the very successful Meyerhoff program for minority students, primarily African-American, for which over 90% of a yearly entrance number of more than fifty go on to graduate school in science or engineering. Many of these students are computer science majors that have had the elective graduate course of Service Oriented Computing for Scientists and Engineers.

? DESCRIPTION (provided by applicant): Glaucoma is a leading cause of irreversible blindness and disability. The disease can remain relatively asymptomatic until late stages and, therefore, early detection and monitoring of functional damage is paramount to prevent functional impairment and blindness. Detection of functional loss in the disease has traditionally been made using standard automated perimetry (SAP). However, SAP testing is limited by the complexity of the examination, subjectivity of patient responses, large variability, cost, and lack of portability. The overall goal of this proposal is to address limitations of currently available techniques by developing a portable objective method for assessment of visual field loss in glaucoma. The investigations of this proposal will address the following 3 specific aims: 1) To develop a portable, objective, multifocal steady state (mfSSVEP)-based visual field assessment platform, integrating a wearable, wireless dry EEG system and a head-mounted display; 2) To develop and validate an Electrooculogram (EOG)-guided method to assess eye- gaze during testing with the envisioned portable platform and 3) To evaluate the reproducibility of the envisioned platform and to conduct preliminary studies evaluating its ability to detect visual fiel loss in patients with glaucoma compared to healthy control subjects. In Specific Aim 1, we will develop a prototype portable device integrating a dry-electrode EEG platform to a cell phone-based head-mounted display for stimulus presentation. We have used similar technology for recent development of brain-computer interfaces, with wireless SSVEP data acquisition and processing. In our preliminary investigation, we have also shown the feasibility of using mfSSVEP for assessment of visual field loss. In the current investigation, advanced signal processing methods will be used to improve signal-to-noise ratio of mfSSVEP from high-density recording. Exploratory studies will be conducted by varying test parameters until a stable testing platform is achieved. In Specific Aim 2, we will integrate an EOG method to the portable plataform, in order to identify fixation losses and allow identification of unreliable mfSSVEP signals to be removed from further analyses. Appropriate eye fixation is essential in order to ensure matching of SSVEP signals to corresponding visual field locations. We will conduct experiments to assess whether the EOG method to filter out unreliable signals improves the accuracy of mfSSVEP to detect visual field losses. In Specific Aim 3, we will conduct validating studies evaluating the reproducibility of measurements obtained with the proposed device, as well as its accuracy for detecting visual field loss in patients with glaucoma. A validated, portable, objective method for assessment of visual field loss in glaucoma may potentially improve screening, diagnosis and detection of glaucoma progression and reduce rates of functional impairment and blindness from the disease.

Electrophysiology has long played an important role in our understanding of brain dynamics in learning, but only at an individual level. With the advent of low-cost, easy to use electroencephalography (EEG) devices that can measure brain responses from an entire classroom at a time, it is now possible to study group brain dynamics during learning in a naturalist classroom setting. This is opportunity to gather data from thousands of students and schools will provide insights not available from earlier approaches. There is however, a critical need to rigorously define how this technology might play a role in improving learning. To address this need, the present project will form a research network of experts to develop and test new EEG methods to measure group brain dynamics. This network includes experts in child development, learning EEG technology, classroom teaching and outreach. Collaborative research will be conducted in three geographic areas reflecting diverse populations: San Diego, the San Francisco Bay Area and the Washington DC Area. The research will focus on assessing important foundational skills: using EEG to assess the quality of encoding of speech by the brain, using EEG to assess attention in real-time, and examining the role of temporal synchronization in promoting attentional behaviors. In addition to research the project features a strong commitment to building interdisciplinary connections within science, as well as vigorously engaging the study communities in the research to foster productive dialog among educators, students, and other researchers.

EEG techniques have made important contributions to understanding dynamic processes in learning, attention, prospection, memory, and executive function. Newer measures, such as the complex Auditory Brainstem Response (cABR) reveal how the brain encodes sound, providing a powerful tool to link brain function to reading and language impairment. Traditional EEG methods are too cumbersome and intrusive for large- scale classroom use. Network researchers have developed a transformative low-cost, dry-electrode, high sampling rate EEG system that is ergonomic and suitable for children. Their software innovations, which enable the synchronization of large-group EEG recordings, will support data recording from twenty+ students simultaneously. These will contribute to at least two innovations in education-based research: 1) It will be possible for a large number of traditional, individual EEG tests to be conducted in parallel, vastly increasing study efficiency, yield and power, and 2) the more revolutionary step of assessing individual and group brain dynamics in ecological learning environments to obtain real-time insight into the dynamics of learning, speech comprehension, and attention. Four experimental goals have been proposed: 1) Identifying Individual Learning Differences with cABR, 2) Developing measures of nested hierarchical processing of speech, 3) Developing protocols for individual and group cognitive state (attention, alertness) assessment, and 4) examining how brain dynamics at a group level reflect speech comprehension in a classroom setting. In addition to its scientific goals, the network will foster scientific dialog by hosting a Workshop on "Brain Dynamics in Learning" to develop a roadmap for future research in the field.

The increasing prevalence of learning disorders, attention deficits, and lackluster appetite for reading across all walks of life, and particularly among school-age children, poses severe problems to humanity and, in the long run, burdens social and economic development. This Partnership for Innovation Building Innovation Capacity (PFI:BIC) collaborative project tackles the impending threats to humanity of illiteracy and faltering education heads-on by creating a new smart-service human-computer interface (HCI) neurotechnology platform as a highly effective, user-friendly, and fun-to-use tool aiding learning and stimulating cognitive development at home and in the classroom. The immersive HCI neurotechnology will allow directly measuring progress at the cognitive level and providing real-time feedback to guide the user in learning to read more effectively. The project is highly Science, Technology, Engineering and Mathematics (STEM) intensive both in its activities and in the targeted benefits of the developed technology, which extends directly to learning science and mathematics by probing cognitive performance of children while they solve puzzles. The development of unobtrusive neurotechnology further addresses a critical need for practical integrated and modular brain-computer interface (BCI) solutions in HCI promoting widespread consumer and clinical use in the marketplace. The partnership provides opportunities for students to gain practical experience in innovation in the marketplace through internships with the industrial partners.

The central aim is to develop and leverage new HCI technology as a learning coach and personal cognitive development assistant that facilitates learning to read and acquiring other critical skills in cognitive development. The immersive yet unobtrusive HCI technology testbed will comprise a dry-electrode electroencephalography (EEG) BCI, a tablet with touchscreen and integrated camera, and a suite of signal processing algorithms running in the cloud, for monitoring brain and gaze activity in children learning to read, and providing real-time neurofeedback on progress in cognitive performance to promote enhanced learning. The partnership will transition scientific advances of a previous NSF-sponsored UCSD project (NSF EFRI-M3C, ENG-1137279) in studying the distributed dynamics of human motor control, to development of neurofeedback training paradigms for learning enhancement, and to practical deployment on the unobtrusive immersive testbed implemented using Cognionics dry-electrode EEG wireless BCI neurotechnology and Syntrogi real-time cloud-based signal processing software pipelines. The potential for human empowerment by the technology will be demonstrated by evaluating effectiveness in enhancing learning capabilities and cognitive performance in simulated classroom settings and other targeted learning environments.

The lead institution for the project is University of California San Diego (UCSD), with investigators from the Institute for Neural Computation and Department of Bioengineering. The industrial partners in the effort are Syntrogi Inc. (dba Qusp, small business, San Diego CA) and Cognionics, Inc. (small business, San Diego, CA). The project also engages broader context partners Drs. Andrea Chiba and Leanne Chukoskie from the UCSD Temporal Dynamics of Learning Center, Dr. Barbara Moss from San Diego State University Department of Psychology, and Dr. Zewelanji N. Serpell from Virginia Commonwealth University Department of Psychology, in the human case studies and the assessment of the developed HCI technology in diverse learning environments.

This project examines how variability in daily stress and fatigue--previously dismissed as uncontrollable 'noise' in cognitive processing--relates to variability in learning and problem-solving. This project is motivated in part by evidence that performance on tests of executive function and memory can fluctuate as a function of recently experienced, real-world, daily stressors. Because problem-solving recruits these and other cognitive abilities, it is hypothesized that day-to-day stressors can also impact our approaches to and success with complex, open-ended challenges regularly faced in educational and professional contexts. Further, on the basis of preliminary research, it is anticipated that a subset of individuals will exhibit greater resilience to stressors than others. In other words, for some, day-to-day stressors may create contexts that facilitate tackling problems to greater or lesser degrees, whereas for others, recent stressors may impact outcomes minimally. This work wields important implications for STEM education given the increasing priority placed on problem-solving skills. It will offer new foundations for modeling individual differences in resilience and vulnerability to everyday stressors during complex tasks. Moreover, understanding stressor-related intra-individual variability can lead to strategies for improving performance of high-stakes, resource demanding operations (e.g., piloting an airplane).

Building from methods of Ecological Momentary Assessment (EMA), both subjective and physiological measures of stress and fatigue will be sampled from healthy adults on a daily basis as they engage in their regular routines of daily life. These data will be uploaded by participants to a secure server via smart phone and will be monitored by research staff. When daily sampling logs suggest the recent experience of high, medium, or low levels of stressors, participants will be scheduled for a testing session to be conducted in their own home or at the research facilities of the Swartz Center for Computational Neuroscience. They will engage in STEM-related problem-solving tasks modeled after real world activities. Simultaneously, electroencephalographic (EEG) data, eye movement, and electrocardiography will be recorded and synchronized. Monitoring periods are expected to last between one and three months and encompass nine testing sessions. This project will result in a rich corpus of data that can be probed from many different angles, offering an unprecedented view of intra-individual variability in task performance as a function of day-to-day changes in physiological and cognitive state. It is expected to reveal behavioral and brain dynamics supporting insight and discovery. Further, expanding from episode-based models of the metacognitive components of problem-solving, it is expected to tease apart ways in which various theorized components--such as representing the problem space, exploring, planning a solution, and implementation--may be differently affected by daily stressors. This project is funded by Integrative Strategies for Understanding Neural and Cognitive Systems (NSF-NCS), a multidisciplinary program jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).

UCSD and FDA will jointly develop and test a wearable and easily deployable system, based on virtual-reality goggles, to collect synchronized brain and biometric signals from mild traumatic brain injury (TBI) patients and healthy participants. They will also develop advanced computational approaches to identify abnormal biomedical signals associated with TBI. The goal of this project is to test the feasibility of using a portable, compact, and deployable system as a pre- and post-hospital diagnostic and monitoring tool of TBI.


UCSD and FDA jointly propose an intense, twelve-month program to design, develop, fabricate and test a wearable and easily deployable system, based on a Virtual-Reality (VR) Head-Mounted Display (HMD), to collect synchronized neural and physiological data from TBI patients and healthy participants. The use of VR HMD allows the use a compact form factor for collecting multi-modal bio-sensing data in a comfortably wearable manner, providing unprecedented opportunities to assess and evaluate the brain and mind under various cognitive tasks immediately after an injury (i.e. pre-hospital care)
The objective of this project is to determine the utility of a portable and multi-modal physiological signal monitoring system as a diagnostic and monitoring tool of TBI.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.