David Brang - US grants

DESCRIPTION (provided by applicant): The proposed research seeks to identify the anatomical and functional networks that support the facilitation of auditory speech processes by visual information. This research line is built on a novel model describing low- and high-level multisensory mechanisms that mediate speech perception processes and is evaluated by anatomical and functional measures of connectivity in typically developing individuals and patients with epilepsy. These aims will be accomplished through a detailed career development plan outlining my training in functional measures of connectivity using electroencephalography (EEG), intracranial electrocorticography (ECoG) in patients with epilepsy, as well as training in rigorous research on speech perception and epilepsy. This training regiment is supported by a team of respected scientists who together have expertise in each aspect of my research and career development plans. Drs. Suzuki and Grabowecky have expertise in auditory- visual multisensory processes including speech perception. Dr. Paller is an esteemed EEG and MRI researcher who has an established record of successfully training graduate students and post-doctoral researchers who advance to tenure-track positions at top Universities. Dr. Leo Towle from the University of Chicago is an established epilepsy researcher and neurophysiologist who uses intracranial ECoG measurements to study memory and language processes. Furthermore, each of these mentors has an excellent track record of administrative and leadership experience: Dr. Grabowecky as former assistant Chair of the Psychology Department, Dr. Paller as the Director of the Cognitive Neuroscience program, Dr. Towle as former Chair of Neurology and President of the American Society of Neurophysiologic Monitoring, and Dr. Suzuki as the Director of the Cognitive Division of the Psychology Department. Training from my mentors will be supplemented by attending four technique-based workshops, four academic conferences each year, as well as weekly meetings with each mentor and scholarly meetings at Northwestern University and the University of Chicago to discuss recent research. The overarching goal for this proposal is to advance my understanding of multisensory and speech related processes in order to conduct leading health-related research and to obtain a tenure-track position at a top research university. Supported by a thorough training regiment, this proposed research will provide a thorough understanding of the mechanisms that support multisensory speech perception and will provide clinically relevant understanding of multisensory processes as a compensatory mechanism.

This project aims to advance scientific knowledge about human vision and use neuroscience to enhance artificial intelligence for computer vision. Vision is central to how humans see and explore the world. About a dozen brain regions work together to process visual information within a fraction of a second. It is hypothesized that these brain regions actively predict one's visual surroundings and use errors of prediction to update their internal representations and guide actions. However, it is not clear how the brain performs computations for recognition and prediction, and whether it is possible for a machine to mimic the brain and recognize and predict visual input in complex, noisy, and uncertain circumstances. This project will address these questions from computational, psychological, and neuroscientific perspectives and deliver new models, data, and tools that promote the synergy between artificial intelligence and neuroscience.

Investigators will design a model based on predictive coding in the brain, and test its ability to perform computer vision tasks and explain human behaviors and brain responses to naturalistic visual stimuli. The investigators will first develop a deep neural network referred to as the predictive coding network. Unlike existing feedforward neural networks, the currently predominant vision models, the predictive coding network has several defining features relevant to neural processing in the brain. It is bi-directional, processing information both bottom-up and top-down. It is recurrent, utilizing the same architecture for dynamic computation. It is parallel, allowing information processing to occur in parallel both within and across different layers. It is both discriminative and generative, reconciling image recognition and synthesis in a single framework. The predictive coding network will be evaluated against benchmark data sets. It is hypothesized to reach competitive performance with many fewer parameters than the state of the art. Then, the investigators will test the model's behaviors given naturalistic images degraded in various ways and/or presented for various durations. It is hypothesized that the model will be more robust and accurate after running for increasingly longer times and reach a time-accuracy tradeoff like human perception under similar conditions. To test this hypothesis, the investigators will perform human behavioral experiments and compare the model's behaviors against human behaviors. Further, the investigators will test the model's ability to explain brain responses to naturalistic images and videos, measured with functional magnetic resonance imaging and intracranial electroencephalography. The model is hypothesized to be able to predict the brain's dynamic activity and representation given naturalistic stimuli. The successful completion of this project is expected to deliver a brain-inspired vision model learnable and computable end-to-end. This model will empower machines with adaptive and robust vision and provide a tool for understanding the computational basis of biological vision.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.