Murat Akcakaya, Ph.D. - US grants

Spatial presence, in Virtual Reality (VR) terminology, refers to the perception (or illusion) of being physically present in a simulated environment. VR strives to create interactive environments that provide experiences of spatial presence through accurate delivery and perception of multimodal sensory stimuli. Research in VR spans fields ranging from neuroscience and medicine to gaming. While the computing and gaming industries have generated tremendous advances in hardware and software for graphics processing and 3D display technologies, VR systems still lack capabilities for providing users with haptic feedback (a sense of touch), which is crucial for generating truly immersive, real-world experiences. It is known that an increase in the feeling of spatial presence manifests itself in the form of increased brain activity. This research aims to achieve the control of haptic sensory stimulation adaptively, based on the changes in brain activity associated with perceptual responses elicited by sensory stimulation in VR environments. Project outcomes will include novel scientific discoveries and engineering enhancements that will make significant contributions to other areas of interest, such as prosthetic limbs, augmented reality, and telepresence applications. The project will help train a new generation of engineers skilled in addressing multidisciplinary challenges, while through outreach activities STEM careers will be promoted at the K-12 level.

The research objective is to identify and analyze brain activity associated with the increased feeling of haptic spatial presence elicited by electro-tactile stimulation and measured through EEG, and to investigate closed-loop techniques to control electro-tactile stimulation for enhanced haptic presence in VR environments. Specifically, the project will: (1) develop an electrical haptic stimulation framework; (2) design analysis techniques to identify markers of haptic inputs in EEG; (3) establish control policies for adaptive electrical stimulation; and (4) evaluate and refine EEG-guided adaptive stimuli control framework in VR environments. In particular, the proposition to actuate haptic feedback through electrical stimulation is novel, while formulating design principles for model-based optimal EEG-guided closed-loop haptic feedback for immersive spatial presence is transformative. Additional innovative propositions to advance adaptive control under uncertainty and psychophysical investigations are unique; these present a potentially game-changing opportunity for VR system development and perhaps for general human-computer interaction.

Unilateral spatial neglect is a perceptual disorder that is one of the most common consequences of right-side brain damage after stroke, occurring in 29% of the 15 million people who sustain stroke worldwide. Patients with neglect demonstrate inattention to objects or events on the side that is opposite to the damaged part of the brain. They often miss food on one side of the plate, missing words on one side of the page, bumping into the left door jamb, getting confused by moving objects, and being fearful of walking in crowded places. The current gold standard for detecting and rehabilitating neglect lacks generalizability to dynamic tasks and contexts encountered during activities of daily living (ADL). The investigators in this project will develop a brain-computer interface (BCI) system that will be implemented in augmented reality (AR) environment for detection, assessment and rehabilitation of unilateral neglect during ADL. More specifically, the system will in real-time monitor the brain activity recorded through electroencephalography (EEG) for the detection and assessment of visually neglected extra-personal space. Moreover, the system will also include haptic, auditory and visual stimulation while the users are engaged in real-world tasks conducted during rehabilitation for reducing neglect-related disabilities. It is also anticipated that the novel scientific discoveries and engineering enhancements of this project will have effects on the current practice on BCIs: (i) enabling design and implementation of such systems in more naturalistic environments providing more immersive experiences; and (ii) expansion of the use of BCIs in the design of intervention and rehabilitation techniques for other neurological disorders. This project will promote STEM education and provide rigorous training and variety of hands-on experiences to researchers from K-12 to graduate level.

The research objective of this project is to introduce a prototype for stroke-induced neglect detection, assessment, and rehabilitation system, featuring: (i) seamless integration of EEG and AR in the design of visually evoked EEG-based BCIs to operate during activities of daily living; (ii) accurate and continuous EEG event related potential detection for neglect assessment and mapping through Bayesian inference models; (iii) information theoretic optimum design of neglect intervention focusing on activities of daily living; and (iv) multimodal real-time feedback for rehabilitation of neglect related disabilities during intervention. Unlike the common computerized neglect assessment methods, EEG will not require any physical responses from the patient. Also, the use of EEG permits automation, making it an ideal method to guide a personalized and automated neglect intervention. It is known that one common element among the existing interventions that have shown promise for reducing neglect is multimodal stimulation to the neglected side of the body or environment. Timely feedback to the user when neglect is detected during the continuous EEG monitoring will enable this stimulation. Finally, used in conjunction with AR headset and skill-based training during acute inpatient rehabilitation, the planned system will provide the opportunity to deliver high-intensity repetitive stimulation with progression during meaningful everyday activities. The outcomes of this project will be disseminated to the scientific community through technical reports, journal publications and conference presentations. All software developed through this project will be publicly available through archival repositories.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.