Jeffrey M. Yau, Ph.D. - US grants

PROJECT SUMMARY/ABSTRACT Temporal frequency is a fundamental sensory domain that is critically important to how we communicate (e.g., speech processing by audition) and interact with objects in our environment (texture processing by touch). Our auditory and tactile senses redundantly signal temporal frequencies spanning tens to hundreds of cycles per second. This overlap enables audition and touch to interact, which can be beneficial because the information available by combining across independent sensory cues is more accurate than that provided by a single sensory cue. Despite this background, we do not have a clear understanding of the relationship between auditory and tactile frequency processing mechanisms. Previous investigations of the neural substrates supporting audition and touch have traditionally focused on a single sensory modality. Here we will test the hypothesis that common brain regions and neural mechanisms, termed supramodal, support auditory and tactile frequency processing. We will develop a computational model of how sensory neurons may combine auditory and tactile frequency information and how these neurons may be changed by adaptation. We will compare the model's predictions to behavioral data acquired in human psychophysical experiments. Using blood oxygen-level dependent functional magnetic resonance imaging (BOLD fMRI) and sensory adaptation, we will localize brain regions whose response patterns are consistent with neural adaptation, and we will use this approach to test whether brain regions represent (i.e., adapt to) both auditory and tactile frequency information. Using fMRI and multivariate pattern analysis (MVPA), we will identify the brain regions from which auditory and tactile frequency information can be decoded. We will determine whether regions support decodable frequency representations for both senses. Our preliminary modeling, psychophysics, and imaging results suggest that multiple regions in perisylvian cortex, including areas classically defined as unimodal, display frequency-selective responses to both auditory and tactile stimulation. This pattern suggests that perisylvian areas may serve as a supramodal network for frequency processing. We hypothesize that attention to vibration frequency enhances the functional connectivity in this frequency network. We will causally probe functional connectivity between somatosensory cortex and auditory cortex by combining transcranial magnetic stimulation (TMS) with fMRI (in concurrent TMS-fMRI experiments) and behavior (in psychophysical experiments). According to our hypothesis, we predict that neural changes caused by TMS of somatosensory cortex should propagate to auditory cortex when subjects attend to vibration frequency. This propagation should modulate auditory cortex activity and auditory perception. These predicted results would support the notion that classically defined somatosensory and auditory areas collaborate to process temporal frequency information as a supramodal network. Supramodal networks may support other fundamental sensory operations like shape and motion processing.

In order to use our hands to manipulate objects, the nervous system must be able to coordinate and interpret a large amount of disparate sensory information about the position of our hands, the arrangement and positioning of the fingers, and what each is contacting. The computations are complicated by the fact that the information from the two hands must be interpreted relative to their spatial position. For example, the left and right index fingers occupy adjacent spaces when the hands are held together but the fingers can occupy vastly different spaces when the arms are spread apart. In these different situations, we can conceivably rely on our sense of vision to help interpret our sense of touch. Vision can provide information about where our hands are in space as well as what objects we touch and manipulate. Despite the obvious importance of vision for interpreting touch information from the hands, the specific ways in which the visual influence occurs remains poorly understood. This research is aimed at investigating the effects of simple and complex visual cues on touch information from the two hands in bimanual tasks. Understanding bimanual touch processing will facilitate effective nonverbal physical communication between people and intelligent machines and it will support the development of highly dexterous robotic systems, advanced neuroprosthetics, and sensorimotor rehabilitation strategies. Establishing how vision influences our perception of our body in real and virtual environments has clear implications for society?s use of immersive and interactive technologies. The project will also promote the participation and education of young women in neuroscience and related STEM fields.

This research program uses rigorous psychophysics, virtual reality technologies, and computational modeling to elucidate the effects of simple and complex visuospatial cues on the perception of bimanual touch. Behavioral experiments will be performed to quantify the effects of simple light flashes on tactile perception and learning. Neural network models will be fitted to the behavioral data to make inferences about how the nervous system mediates visuotactile and spatial interactions and how neural circuits may be modified through learning. The effects of complex visual cues on proprioception and bimanual touch will also be examined by performing psychophysics experiments in virtual reality. Specifically, visual proprioception cues and virtual object information will be altered in a virtual reality design, allowing the investigator to test hypotheses regarding multisensory integration for proprioception and causal inference processing. Collectively, the research program will yield novel insight into how visual information modulates the perception of tactile cues experienced over the two hands.

Co-funded by the M3X (Engineering) and PAC (Social, Behavioral, and Economic Sciences) Programs

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.