The goal of this project is to build and evaluate a system for exploring how the human brain processes information in everyday real-world environments. The investigators will directly record from the brain while participants navigate the real world, while synchronously recording information about the participant's first-person experience from a set of sensors including cameras, microphones, eye-tracking, and physiological recordings. Neurosurgical participants with a clinically implanted neural recording and stimulation system volunteer for these experiments, providing rare direct human brain recordings as they move around a real-world environment. The rich sensor data captured from the participant's first-person experience will be analyzed in relation to the neural data to infer how changes in patterns of neural activity over time relate to changes in experience. In addition, stimulation will be applied, at safe levels and timed according to "event boundaries" of the participant's experience, to determine whether memories of specific events can be enhanced. The proposed platform that allows for neural recording, direct brain stimulation, and synchronization with external, wearable devices will open an entirely new area of research at the intersection of computer science, engineering, cognition, and clinical neuroscience. These studies will launch and accelerate an emerging and pivotal area of research that will provide therapeutic interventions, proven in the real-world, for participants afflicted with debilitating cognitive disorders. This project will also make substantial contributions to education and outreach, including the development of K-12 classroom modules, interdisciplinary graduate training, outreach to industry partners in the neuromodulation field, and workshops at local Salt Lake City memory care communities.
Development of this neural and first-person experience recording system will entail three collaborative research tasks: i) Synchronizing the Human Experience Relative to Neuronal Events: This module will develop a robust framework to record and synchronize neuronal activity along with internet of things (IoT) sensor data representing a broad subset of human sensory channels. The design will be portable such that the human experience can be reasoned about outside of a simulated lab environment. ii) Real-time Semantic Alignment between Human and IoT Perception: Reasoning about the complex relationships between neural biomarkers and the human experience captured by IoT sensing requires more than sensor synchronization. Neural-symbolic approaches that integrate the perception capabilities of deep learning with human logic will be leveraged to reason about the high-level complex spatiotemporal dependencies across a heterogeneous set of sensors. iii) Enhancing Episodic Memories of Real-world Experiences with DBS: Given a proper characterization of neural oscillations associated with event boundaries, the investigators will work to enhance episodic memories of real-world experiences with wireless deep brain stimulation (DBS) devices by directly stimulating the human brain. Under medical supervision, stimulation will be applied to the human amygdala at and between event boundaries in subjects with implanted stimulation devices as they encounter novel, 3D augmented reality objects while navigating a large-scale, real-world environment. Memory will be subsequently tested in laboratory and real-world settings. These three research areas will develop a situational understanding of neuronal activity in the context of human experience. They will further lay the foundation for future research directions in safe and effective stimulation of the brain in response to human experience in the wild.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
