Shinsuke Shimojo - US grants

This research uses a striking visual illusion, the flash-lag effect, to study the normal functioning of the human visual system. One example of the illusion involves a line segment rotating about its center. A stationary segment is briefly flashed at each end of the rotating segment, such that the rotating and stationary segments are in alignment. Surprisingly, observers perceive the stationary segments not as aligned with the rotating segment, but rather as lagging slightly behind it. Even more striking is a version of the phenomenon involving color. If a red line is flashed within a stationary green bar, observers perceive the line as yellow (because mixing red and green light yields yellow, due to the way the visual system processes color). However, if a red line is flashed briefly within a rotating green bar, observers don't see a yellow line within a green bar; rather they see a red line trailing the green bar. Our hypothesis is that the flash-lag effects result from mechanisms within the visual system that compensate for delays in neural transmission time. Transmission of information from the eyes to the brain takes appreciable time; by the time information about a moving object reaches the brain, the object has moved some distance. Thus, we might expect the object to be seen not where it actually is, but rather where it was a short time ago. However, we suggest that the visual system extrapolates the registered location of smoothly moving objects to compensate for the effects of the delay. Thus, given collinear moving and stationary line segments, the moving segment is seen as leading the stationary segment because the visual system extrapolates the location of the moving but not the stationary segment. We will study several forms of the flash-lag effect under various conditions. The results will increase our understanding of how the visual system compensates for neural delays; also, the studies involving colored stimuli will allow us to test competing theories of color vision.

Visual scenes do not appear to shift when an eye movement causes the optical image to move across the retina, but do when the same retinal displacement occurs while the eyes are stationary. How does the visual system achieve this "position constancy" during eye movements? The goal of this project is to understand the coordinate transformations which enable position constancy by generating a body- or environment-centered description of objects from their eye-centered representation. Instead of the traditional approach to this problem, measuring mislocalization of a briefly flashed dot stimulus, we will examine various types of aftereffect/cue effect, including visual aftereffects (eg. figural, and motion), visual-motor aftereffects (eg. saccade. and arm reaching) and an attention-induced motion illusion (the "shooting" line motion effect) by employing what we call the fixation/refixation paradigm. Focusing on such effects, we could investigate the coordinate transformation process which is relevant to the overall spatial context, and thus higher-order and biologically significant. The typical experiment will consist of an adapting (cueing) phase, and a testing phase to measure negative aftereffect (or strength/direction of the illusory motion). Unlike the conventional procedure, however, there will be a refixation phase between the adapting and the testing in which the fixation point will move, and the subject will be asked to make either a saccade or smooth pursuit eye movement to it before the test presentation. This manipulation enables us to isolate the adapted location in coordinate systems or maps, including the eye-centered and the non-eyecentered, i.e. the head-, the body-, and the environment-centered. By presenting the test stimulus at various locations, we will obtain a spatial tuning curve of the effect and apply a cross-correlation. analysis to find out the degree to which the visual system maintains the eye-centered representation, and/or transfers it into a non-eye-centered description. If we find a significant non-eye-centered component, as indicated by our pilot studies, we will determine exactly which non-eye-centered system (head-, body-, or environment-centered) is relevant by rotating the observer's head and/or torso in a vestibular chair. We will further compare active and passive head/body turn conditions to identify the nature of extraretinal signals on head position which contribute to this transformation process. The results will bridge the gap between animal physiology and human psychophysics, and provide insights into-the processes underlying coordinate transformations.

DESCRIPTION: Most perceptual, cognitive, affective, and linguistic events are specified concurrently in different sensory modalities and are distributed across space and time. Spatial and temporal stimulus features can be represented equally well across modalities and thus provide a major basis for the integration of the heteromodal attributes of multimodally represented events and objects. Empirical evidence indicates that human infants can perceive some of these intermodal attributes and can use them to integrate heteromodal inputs. Most of this evidence concerns infants' perception and usage of temporal aspects of multimodal stimuli, such as synchrony, duration, rate, or rhythm. There has been relatively little research on the role of spatial factors in crossmodal integration, and the results remain equivocal in many cases. Given that spatially integrated multimodal events are a fundamental part of the infant's everyday experience, understanding the processes underlying the development of such abilities is critical. The purpose of the current project is to carry out a systematic investigation of the development of infants' responsiveness to intermodal spatial relations by studying responsiveness to intermodal targets with varying degrees of spatial separation in infants between 2 and 10 months of age. Head- and eye movements will be measured and will be used to investigate developmental differences in infants' responsiveness. A series of seven experiments is proposed to test various aspects of auditory-visual localization. The experimental apparatus has been designed in accordance with previous research of A-V localization in human adults and animals. The latter research will thus provide important baseline data and constraints of for interpreting the results obtained in this project. Standard psychophysical methods (race model analysis) will be employed to establish the mechanisms underlying the multimodal spatial integration process and its development. The empirical work will be complemented by modeling work. The model will incorporate the empirical results, and will provide a framework for the empirical research. The results will: (a) explicate mechanisms underlying the developmental changes in responsiveness to intermodal spatial relations, (b) add to our understanding of the development of a process that enables infants to learn about the psychological unity of their experience, and (c) help further develop and refine measures of perceptual functioning that already have proved to have diagnostic utility in detecting aberrant developmental outcomes.

Collaborative Research: Development of Transcranial Magnetic Stimulation Coils for Cognitive Neuroscience Research
Abstract

With National Science Foundation support, Drs. Shimojo, Gabrieli, Maeda, and colleagues will conduct a two-year project aimed at developing and validating a new transcranial magnetic stimulation coil device. A transcranial magnetic stimulator (TMS) coil produces relatively focal magnetic current which can be used non-invasively (outside the skull) to safely create "virtual brain lesions" and to transiently modulate neural activity. It can probe brain areas at different points in time during the performance of a psychological task and thereby assess the causal relationship between brain activity at the targeted location and components of task processing.
It is particularly powerful in combination with neuroimaging, which can provide activation maps and precise stereotactic reference frames (anatomical MRI) for accurate and task-relevant TMS targeting. TMS provides a means to study of causal relationships between brain activity and behavior by induction/facilitation or extinction/inhibition of an effect, a significant advantage over other brain imaging techniques that in general offer only correlational effects. The device to be developed in this project will allow rapid switching of current in the TMS coil in a novel fashion that allows much greater range of precisely matched control conditions, and it is particularly useful in neuroimaging settings.

More specifically, the proposed device will deliver three kinds of stimulation without having to move or switch coils, and all while causing similar sensations hence being unnoticed to the participants. First, I can produce real TMS. Second, it can produce sham TMS: by manipulating the current in each loop of the coil, most of the induced electric field will be cancelled out, hence resulting in no substantial physiological effect. Third, the device can produce real TMS with reversed current direction. Many brain regions have optimal TMS current direction to disrupt or evoke an effect, and by reversing the current direction in the coil, this type of TMS applies non-optimal stimulation while maintaining identical subjective sensation and induced electric field. In many cases, the coil can be set so that this reversed current is ideal to stimulate another (usually the contralateral) site. The device will be validated using well-established tools and paradigms in the fields of cognitive neuroscience and TMS. In particular the proposed device will be validated with respect to (1) its physical properties using a search-coil electric circuit, (2) associated subjective sensations, (3) in behavioral paradigms of motor physiology and (4) visual psychophysics, and (5) electroencephalographic (EEG) recordings of the TMS-evoked responses and behavior/TMS interaction.

Intellectual Merit. This proposed TMS device is superior to conventional TMS in five ways. First, it can be used in fluently and simultaneous with brain imaging studies in which moving or switching coils is highly undesirable. Second, it can be used to interleave different sham and control conditions with great control. Third, it can be used in a double blind designs, in which neither the experimenter apply the TMS nor the subject know whether the TMS is real, sham TMS, or real-TMS in the reverse direction. Fourth, the device can be used to produce consecutive and controlled activations of either the same brain area or two different neural structures, without the participant's awareness. Fifth, it can be used to investigate the physiologic and behavioral interactions of two consecutive stimuli in the same or different current directions.

Broader Impacts. Undergraduate and graduate students, postdoctoral fellows, and staff members will receive interdisciplinary training in psychophysics, electrical engineering, neurophysiology, and signal processing. The proposed device will be used by cognitive, clinical, and TMS neuroscience research communities to enable new avenues for more sophisticated and novel research techniques for better understanding of brain function, pathophysiology of neuropsychiatric disorders, and the mechanism of action of TMS. Findings and techniques will be disseminated through conferences, journals, and our laboratory web-sites. Circuit diagrams, theories, and results of our validation studies will be available at those web-sites. Upon success of this proposal and with demand from the research community, this device should become commercially available from a major manufacturer.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

This NSF Major Research Instrumentation (MRI-R2) Award will enable a three-year grant to purchase an upgrade for a single piece of equipment for imaging the human brain at the California Institute of Technology. The upgrade, a 32-channel Total Imaging Matrix upgrade of a Siemens 3.0 Tesla MRI scanner, will substantially improve the resolution, the speed with which experiments can be done, and the kinds of imaging sequences that can be programmed. Taken together, these major enhancements will enable a range of questions about the structure, connectivity, and functioning of the human brain. Researchers at Caltech, in collaboration with a national and international consortium of scientists, will use the equipment to investigate how the brain makes financial decisions, how social information such as faces are processed, and how brain-machine interfaces can be built to decipher information from the brain to guide robotic prostheses. These are important, big open questions in neuroscience, and the new equipment will greatly enhance science at the Caltech Brain Imaging Center.

The grant will also provide opportunities for training of students and post-docs on the new equipment. This will include classes taught at Caltech as well as participation in individual research projects. The development of these new scientific tools will lead to a better understanding of how the brain works, how it is wired up, and how it may dysfunction in disease. That knowledge, in turn, will contribute to efforts to build artificially intelligent systems. Taken together, the cutting-edge science enabled by the new equipment, and the training of the next generation of young scientists on it, will contribute substantially to cognitive neuroscience in America and worldwide.

DESCRIPTION (provided by applicant): The long-term goal of this research is to understand the principles that govern crossmodal interactions in the temporal domain. The specific objective of this proposal is to identify and to characterize a supramodal timing mechanism for perception of rate. The central hypothesis is that there is a vigorous and adaptive central timing mechanism (responding to changes within the order of seconds to minutes) that is at least to some degree unified and amodal. The proposed studies will rely on a recently discovered crossmodal aftereffect in which repeated exposure to a pulsed adaptor stimulus leads to changes in the perception of the rate of pulsation of subsequent stimuli, such that a relatively fast adaptor lead to later stimuli being perceived as slower, while a relatively slow adaptor does the reverse. This happens even when the adaptor and test stimuli are presented with different modalities and are never presented together. In a series of psychophysical experiments, this effect will be tested with different configurations of unimodal, bimodal, and crossmodal visual stimuli. These experiments will measure the strength and direction of the aftereffect as it varies with different adaptor stimuli and with different spatial relationships and rate differences between adaptor and test. These experiments will provide for a strong test of the supramodal timing hypothesis. An additional set of psychophysical experiments will use continuous flash suppression to render participants are largely unaware of the visual stimuli and test whether adaptation still occurs. This will determine whether the effect requires conscious awareness (and thus may be largely cognitive) or is at least in part perceptual. A final set of experiments will use EEG to track the range of temporal frequencies and the perception threshold of the effect, via the study of the power spectrum and steady state frequency responses, and will provide an objective measure of the causal relationship between auditory and visual networks in creating the perceptual effect. The results of these experiments will place constraints on neuronal and computational models of temporal processing. Timing deficits have been linked to changes due to normal aging as well as to disorders including dyslexia, ADHD, Parkinson's disease, depression, and schizophrenia. A better model of how the brain encodes time has the potential to enhance understanding of these timing deficits and may have implications for potential clinical or psychological interventions.

The ability to perceive time is innate to humans, but very little is known on the underlying psychological and neural mechanisms. Some peculiar daily life and laboratory phenomena where time appears to be frozen for a brief moment, termed "chronostasis," especially poses a challenge to cognitive neuroscientists. It may also be related to pathological experiences of time lapse in cases such as schizophrenia and drug-addicts. Chronostasis, which usually happens around the hundreds of millisecond range, could occur with or without action, given different perceptual contexts. This particular phenomenon could be a valuable window into how the brain constructs duration perception. With the support from National Science Foundation, Dr. Shinsuke Shimojo and colleagues will use electroencephalogram (EEG) along with a novel method to measure neural activity and its dynamics corresponding to this phenomenon. First, this project will provide further insights into why people perform a challenging task (for example, in sports or games) with concentrated attention sometimes experience "a flow of time." Second, the novel method could lead to a new neural activity index for clinical screening to understanding of mental disorders, because it is known that various types of patients, such as schizophrenics, autistic, and ADHD patients, have problems in time perception. The new knowledge of neural areas and dynamics may also inspire new programs and devices for training or rehabilitation. Third, the conclusions may potentially impact occupations whose performance relies on precise temporal judgments.

Our scientific hypothesis is that duration perception at the hundreds of milliseconds range is mediated by "hierarchical predictive coding," where both familiarity and novelty play a role in modulating perceived duration. The mechanism governing familiarity is likely "repetition suppression," i.e. a relatively automatic decay of neural responses by repetition; the mechanisms subserving novelty detection could be attention orienting. The researchers will isolate these factors with behavioral experiments, and then find corresponding neural correlates with EEG. Specifically, they will use a steady state visual evoked potential (SSVEP) approach to mark the short duration visual stimuli with a tagging temporal frequency. This will give them advantage in subsequent source localization and network connectivity analysis. They expect to discover interactions between low-level and high-level brain areas that co-decide duration perception, which will provide some clues to the ultimate mystery in behavioral neuroscience, that is, how we could experience time lapse, without having peripheral and specific sensory receptors.

Project Abstract One of the major goals of the BRAIN initiative is to develop technologies capable of interfacing with specific neural circuits in the human brain. Ultrasonic neuromodulation (UNM) is among the most significant new technologies being developed for this purpose because it has the potential to non-invasively modulate neural activity in deep-brain regions with millimeter spatial precision. This unique capability would complement existing neuromodulation and imaging techniques in basic and clinical applications. However, despite a surge of interest in UNM, the lack of knowledge about its mechanisms and recent findings of off-target sensory effects accompanying direct neuromodulation pose significant challenges to the use of this technology in human neuroscience. In particular, while most groups working on UNM are racing ahead with device development and applications, we have uncovered a major issue with this technology that must be addressed before it can be used reliably by the neuroscience community: at the ultrasonic parameters used in most UNM studies, ultrasound causes not only direct neuromodulation of the targeted region, but also strong activation of auditory cortical circuits. This significantly confounds the interpretation of UNM-evoked electrical and motor responses seen in animal models, our understanding of efficacious doses and parameters, and most importantly, potential applications in humans. To overcome this issue, we will (1) establish an understanding of the mechanisms and parameters of both direct and indirect effects of ultrasound on neural circuits, (2) identify parameters for maximizing direct modulation, and (3) develop sham stimuli enabling properly controlled use of UNM as a tool for human neuroscience. To tackle these problems, we have assembled a multidisciplinary team of scientists and engineers with expertise in tissue mechanics, acoustics, biophysics, systems neuroscience, and human psychophysics who will use unique experimental approaches ranging from computational models to specially-developed transgenic rodents and human volunteers. If successful, this project will help resolve a key issue preventing focused ultrasound from serving as a reliable, interpretable modality for non-invasive neuromodulation, and lay the groundwork for the development of optimized devices and appropriate controls for widespread use of UNM in the study of brain circuits. 

Project Summary The visual cortex is known to change its functional map and connectivity with other cortical regions following partial or full vision loss, including signi?cant repurposing among the other senses such as audition. What is unclear is whether such crossmodal incursion alters the multimodal pathways and multisensory integration of the various senses. Also, it is not yet known under what vision loss conditions (such as central vision loss with Age-Related Macular Degeneration, peripheral vision loss with Retinitis Pigmentosa, or full vision loss due to numerous causes) multisensory integration is facilitated (or suppressed), and whether these changes vary with retinal location. Such knowledge is critical to develop a more complete theoretical model of multisensory integration; to better evaluate potential for rehabilitation in those with vision loss; to provide a solid basis for the development of advanced retinal prostheses, sensory aids, and sensory substitution devices; and to develop optimal multisensory training and rehabilitation paradigms following visual restoration. To this end, we propose to determine the spatial and temporal characteristics of auditory-visual (A-V) integration in individuals with low vision (Speci?c Aim 1). More speci?cally, we will use a set of auditory-visual illusions as a psychophysical tool to determine the degree of A-V integration in various retinal locations as a function of both eccentricity from the fovea and proximity to regions of visual loss. We also propose to examine the viability of visual processing and crossmodal integration in those with low vision and the late blind by employing both A-V illusions and mental imagery (Speci?c Aim 2). We will determine whether multisensory integration from imagined visual stimuli can integrate with real auditory stimuli in the late blind to change the perceived location of auditory stimuli, including auditory spatial perception in the horizontal plane and in depth. The results from these two aims will provide an assessment of the key characteristics of auditory-visual interactions in the blind and those with low vision, and will identify differences in these multisensory interactions that are speci?c to the cause of vision loss. We also plan to identify the neural correlates of such crossmodal interactions and integrations using fMRI imaging (Speci?c Aim 3). We will examine key differences in visual cortical activity, as well as the connectivity patterns among auditory, visual, and multisensory cortices, by comparing low vision and late blind participants with sighted controls. A comprehensive understanding of the multisensory processing capabilities of low vision and late blind individuals will provide crucial insights into the consequences of functional reorganization in the human brain, and will also pave the way for advanced multisensory aids, visual prostheses, and rehabilitation protocols.