Michele Rucci - US grants

EIA-0130851
Rucci, Michele
Boston Univ.

BITS: Fixational eye movements in biological and artificial vision systems

Summary

The proposed research project has two goals: (1) to determine the specific contributions of small eye
movements to the representation of visual information in the brain, and (2) to export such mechanisms
to machine vision systems. To accomplish these goals, this project will integrate computer modeling of
neural populations, robotic experiments and psychophysical experiments with human subjects to analyze
the statistical structure of neural activity under natural viewing conditions.
The proposed research will first characterize the influences of eye movements on the spatial selectivity of thalamic and cortical cells during natural viewing conditions. Then, the activity dynamics of neuronal ensembles will be studied, both in terms of efficiency of visual representation and in relation to the results of psychophysical studies on the hierarchical structure of spatial visual processing. Finally, we will analyze the possible involvement of small eye movements in the recovery of distance information during binocular free-viewing of a 3D scene. Studies will be conducted with the dual purposes of providing new insights on the brain and considering possible applications to machine vision systems.

Our eyes are never at rest.Even when e are .xating upon a visual target,small involuntary eye
movements continuously perturb the projection of the image on the retina.A substantial body of
recent evidence indicates that these .xational eye movements are an important component of the
ay visual information is acquired and represented in the brain.Neurophysiological studies with
monkeys have sho n that small eye movements strongly a .ect the responses of neurons in the visual
system.In the research of a previous NSF award (EIA-0130851),e have shown that fixational
instability is crucial for identifying stimuli presented for the brief durations that occur during
natural viewing (Rucci and Desbordes,2003),and improves the efficiency of visual representations
in models of the lateral geniculate nucleus and primary visual cortex (Rucci and Casile,2003b,
2004).Furthermore,reproduction of human eye movements in a robotic system has sho n that
.xational instability may provide reliable information of distance (Santini and Rucci,2003).
Building on our previous results and those of recent neurophysiological studies,this proposal
continues part of the research of NSF a ard EIA-0130851.It describes a program of research that
focuses on extracting and integrating multiple depth cues to develop a reliable 3D representation
of the visual scene.This project follows an interdisciplinary approach that integrates computer
modeling of the visual cortex with robotic experiments and measurements of eye movements in
human subjects.The long-term goal of this research is to develop machine vision systems that,by
emulating the computational principles of the brain,achieve high levels of robustness and e .ciency.
The specific aims of this project are to:
1.Develop a model of the monkey primary visual cortex in which,following recent neurophysi-
ological .ndings,distinct populations of neurons respond during di .erent phases of small eye
movements.
2.(a)Analyze the information of relative distance transmitted by di .erent neuronal populations
when the model is coupled with a robotic oculomotor orkstation that replicates human eye
movements;and (b)integrate depth information provided by di .erent cues into a coherent
3D representation of the visual scene.
3.Investigate the self-organization of the model by means of learning,so that the extraction
and integration of depth information is autonomously tuned to the physical and motor char-
acteristics of the system.
In this research,the eye movements of human observers will be measured by a high-resolution
eye-tracker,and then accurately replicated by a robotic system (a head/eye system with t o mo-
bile cameras)that e have speci .cally designed to reproduce the visual inputs to the eyes during
oculomotor behavior.Visual images acquired in this ay will be applied as input to computational
models of neurons in the striate cortex of primates.Intellectual merit: This project establishes
a direct linkbet een human and machine vision studies.By focusing on the computational mecha-
nisms by which eye movements a .ect visual processing,it has the potential of providing new insights
on the brain as ell as opening the ay to the development of new algorithms in machine vision.
Broader impact: The interdisciplinary nature of this research o .ers new opportunities for train-
ing students.By collaborating with the PI,students in the Department of Cognitive and Neural
Systems at Boston University,a department ith an important focus on computational modeling
of the brain,will have an opportunity to combine their theoretical orkwith the development of
machine vision systems.

[unreadable] DESCRIPTION (provided by applicant): During normal viewing, the projection of the visual scene on the retina is never stationary. Even during the brief periods of visual fixation, small eye movements combine with movements of the head and the body to keep the retinal image in motion. Fixational instability may be an important component of how information about a visual scene is acquired and represented in the brain. By affecting the temporal structure of neuronal responses, it may contribute to the encoding of visual information in a way that goes beyond simply refreshing neuronal responses to prevent the fading of unchanging stimuli. Modeling studies that simulated neuronal responses during oculomotor activity suggest specific mechanisms by which this may occur. This proposal combines visual psychophysics and neuronal modeling to examine the influence of fixational eye movements on the identification of stimuli presented for periods comparable to the brief durations of visual fixation. Psychophysical experiments with human subjects will compare identification performances in the presence and absence of retinal image motion and correlate the fine structure of eye movements to visual identification. Experiments will rely both on a traditional technique of retinal stabilization as well as on a new system of gaze-contingent display that will allow selective elimination of eye movements during the periods of fixation following saccades. Neuronal models will simulate the responses of neurons in the retina, LGN and V1. For every experimental trial, a simulation will replicate neuronal activity during the sequence of eye movements performed by the subject. Subject performance and the information about the stimulus transmitted by neuronal firing will be compared on a trial-by-trial basis. A number of visual disorders manifest abnormal fixational eye movements. In addition to advance our basic understanding of visual perception, a comprehension of the functional implications of fixational instability may lead to treatment of the visual impairments commonly associated with such conditions. [unreadable] [unreadable] [unreadable]

Despite the tremendous improvements in computational power of recent years, machine-vision systems are still far from replicating the efficiency, robustness, and speed of biological systems. A critical difference between organisms and machines lies in the acquisition of visual information. Unlike computers, biological vision systems are not passively exposed to the visual scene. Instead, they actively seek useful information by means of goal-directed behavior. It is a long-standing proposal that forging a tight link between behavior and perception may be critical for developing more efficient machine-vision algorithms.

While examining a scene, humans coordinate eye movements with small movements of the head and body. Coordinated head/eye movements provide 3D information in the form of parallax, the different apparent motion of stationary objects at different distances. To examine the impact of this behavior in 3D vision, this project integrates computer modeling of the visual cortex with experiments in robotic vision and human psychophysics. The specific aims of this project are to: (a) measure the influence of coordinated head/eye movements on the accuracy of depth and distance judgments in human observers; (b) measure the 3D information resulting from head/eye movements by replicating human motor activity in an anthropomorphic robot; and (c) model the extraction and the autonomous calibration of the parallax resulting from head/eye movements in the parietal cortex of macaques. By coupling a neural model of the brain with a robot that replicates human behavior, this research establishes a direct link between human and machine vision studies. It has the potential of providing new insights on the brain as well as opening the way to the development of new algorithms in machine vision.

DESCRIPTION (provided by applicant): Our eyes are never at rest, even when attending to a single point. We are normally not aware that, in the periods of fixation, microscopic eye movements continually shift the stimulus on the retina. Visual percepts tend to fade when the stimulus is artificially immobilized on the retina, and it has long been hypothesized that the incessant fixational motion of the eye plays a fundamental role in visual perception. Several findings from our recent NIH-funded research suggest that this motion is, in fact, a critical computational element of an active sensorimotor strategy by which the visual system processes spatial information in the temporal domain. Building upon our recent results, this project examines the perceptual, computational, and neural consequences of using eye movements to represent space through time. It addresses three fundamental questions: (Aim 1) How is spatial information encoded in the modulations of luminance resulting from fixational eye movements? (Aim 2) How is this information extracted and interpreted? (Aim 3) Can the spatiotemporal redistribution of input energy be adjusted according to task by controlling fixational eye movements? To link the perceptual influences of fixational eye movements to their effect on the neural coding of visual information and elucidate encoding/decoding mechanisms, this project integrates visual psychophysics in humans, statistical and computational analysis of retinal input, and neural modeling. Experiments will make critical use of a sophisticated system for gaze-contingent display (already developed and extensively tested), which allows precise, yet highly flexible control of retinal stimulation. Statistical, computational, and modeling studies wil reconstruct and examine the visual input signals experienced by retinal receptors and simulate neuronal responses in the retina and lateral geniculate nucleus. The proposal that the visual system uses behavior to represent space through time challenges current views on the mechanisms of early visual processing at the most fundamental level: it replaces the traditional notion of the retina as a passive encoding stage that optimizes overall information transmission with that of an active, tunable system for feature extraction, whose function can only be understood in conjunction with eye movements. This shift of view implies that eye movements are in part responsible for fundamental properties of spatial vision (e.g., contrast sensitivity an its dynamics) that, at present, are solely attributed to neural mechanisms. Understanding the functional implications of fixational instability may also lead to new treatment approaches for visual impairments in the many disordrs whose manifestations include abnormal fixational eye movements.

The brain is remarkable in its ability to perceive a stable visual world even though the patterns on the retinas change continually as the eyes are always in motion. Voluntary redirections of gaze occur several times per second from one scene location to the next. Even in the intervals between these redirections, when it seems that gaze is fixed at a single location, microscopic eye movements, known as fixational eye movements, continually jitter the stimulus with respect to the retina. Little is known about how the normal alternation of macroscopic and microscopic eye movements affects the retinal stimulus, and about the importance of this dynamic input for the visual system.

With support from the National Science Foundation, Dr. Michele Rucci will examine the influence of eye movements on the acquisition and encoding of visual information. Experiments with human observers will analyze and compare the characteristics of visual input signals to the retina during a number of everyday viewing activities. Computer models of neuronal populations in the early stages of the visual system will be used to investigate the hypothesis that a continually changing retinal stimulus contributes to the establishment of efficient visual representations. This project addresses fundamental questions in visual neuroscience. Pathological conditions which give rise to abnormal eye movements are typically accompanied by visual impairments. Understanding the impact of eye movements on early visual processing is critical to the advancement of knowledge of how the visual system functions in health and disease.

Information that leaves the eye first encounters a structure called the lateral geniculate nucleus (LGN). From the LGN, information is relayed to the cortex, where neurons have properties closely related to our perceptual experience. To understand how the cortex acquires its remarkable abilities, data on its inputs from the LGN are needed. When we view objects in the world, the neurons in the lateral geniculate nucleus (LGN) are stimulated by small regions of space called their receptive fields. This project will record the electrical responses of the LGN neurons of monkeys performing a visual task. Linear mathematical models of the receptive fields of LGN neurons will be constructed based on their responses to simple forms. These models will be used to predict responses of the LGN cells to natural scenes. In addition, the effects of regions of space surrounding the receptive fields will be assessed. These surrounding regions are expected to reduce the activity of the LGN neurons so that they are more efficient and they conserve energy. In addition to improved understanding of the mechanisms underlying human perception, the results may help to generate ideas about improved designs of artificial vision systems that have wide applications in industry. This collaborative project will provide training for a graduate student in neuroscience at the University of Texas and will also provide training and data for graduate students doing mathematical modeling at Boston University.

We are normally not aware that our eyes and head are always in motion. Yet small, involuntary head and eye movements continually occur, even when we attempt to keep our gaze fixed on a point in the scene. It has long been known that the temporal modulations resulting from these movements are essential for seeing: the world becomes progressively fainter and will eventually fade altogether when the head and eyes are immobilized. However, the specific mechanisms by which fixational movements contribute to vision remain unknown. With funding from the National Science Foundation, Dr. Rucci and his team at Boston University will measure microscopic head and eye movements in human observers and reconstruct the input signals entering their eyes. The investigators will then use these data to quantify the impact of the observer's actions on the sensory flow of information and examine how this flow changes during the execution of different visual activities.

This research will investigate the mechanisms by which fixational movements contribute to the processing of visual information in the brain and the establishment of a stable visual percept. Elucidating the impact of fixational movements is critical to advancing our knowledge of how the visual system functions and how to build machines which replicate human visual capabilities. Furthermore, abnormal fixational eye movements
occur in various pathological conditions. A thorough understanding of the functions of fixational eye movements may lead to treatment of the visual impairments associated with such conditions.

How does our visual system transform the light patterns impinging onto the retina into a meaningful three-dimensional world of objects and movement? Over the last few decades we have made enormous progress in our understanding of brain functions. Yet, the fundamental computational and biological principles underlying visual information processing remain elusive. Like other species, we humans do not simply sit back and passively accept the sensory data that flows around us. Instead, we actively seek out useful information by coordinating sensory processing with motor activity on multiple scales. The hypothesis of this project is that visual perception in humans is an intrinsically sensory-motor process, in which the observer's behavior plays a key computational role.

To investigate this hypothesis, the project brings together researchers from the Netherlands (Dr. Brenner at VU University in Amsterdam), the US (Dr. Rucci at Boston University), and the UK (Dr. Rushton at Cardiff University), who have developed critical expertise on the analyses of different types of motor activities in humans. These researchers and their teams will investigate the mechanisms by which people use motor/proprioceptive signals to accomplish visual tasks. Understanding the impact motor activity has on perception is critical to advancing our knowledge of how the visual system functions. Additionally, understanding the interplay between our senses and motor processes will help us to replicate those mechanisms in artificial systems.

In addition to the potential for advancing our understanding of visual perception, this research also has several broader impacts. Many pathological conditions that feature abnormal motor activity also present visual impairments. Understanding the role of behavior in the extraction and processing of visual information could point towards potential treatments for such visual impairments. Furthermore, the interdisciplinary nature of this research offers new opportunities for training students. By collaborating with the PI, students at Boston University will have an opportunity to combine experimental and theoretical research.

Humans are normally not aware that their eyes and head are always in motion. Small involuntary head and eye movements continually occur in the periods between voluntary relocations of gaze, even when attempting to maintain steady gaze on a single point. These movements shift the stimulus on the retina in ways that would be immediately visible had the motion originated from objects in the scene rather than the observer's behavior. Building on a large body of recent findings, this project investigates the hypothesis that fixational head and eye movements are not motor instabilities, but part of a precisely controlled motor strategy that facilitates the visual processing of fine spatial detail. Elucidation of this hypothesis is important not only for advancing knowledge on the fundamental mechanisms of visual perception, but also because it implies that some visual acuity impairments may have unrecognized motor origins, and, conversely, that motor disturbances may have unsuspected sensory consequences. The research could have implications for the optimization of human visual performance in situations of restricted head movement. Also, by shedding light on how to achieve optimal sampling of visual information through coordinated movement, the research could also have implications for the design of artificial vision systems.

This project will follow an integrated theoretical and experimental approach, which combines psychophysical experiments with human subjects, statistical analysis of the stimulus on the retina, and modeling of neurons in the early stages of the visual system to examine the joint consequences of head and eye movements on the acquisition and encoding of fine spatial information. Specifically, the investigators will measure fixational head and eye movements during the normal execution of high-acuity visual tasks. They will study how these movements interact, the degree of control involved in their interplay, and the nature of this interplay. In parallel, the investigators will develop quantitative predictions by reconstructing the visual input signals experienced by the observers' retinas and by analyzing the impact of these signals on the responses of computational models of neurons in the visual system. This synergy of experimental and theoretical methods will result in new knowledge in the fields of sensory perception and motor control. It will contribute critical information on the joint characteristics of fixational head and eye movements, their visual functions, and help in identifying the sensory-motor strategies by which spatial information is represented in the brain.

Augmented and virtual reality (AR/VR) promises to become one of the most disruptive technologies of the 21st century, revolutionizing how we interact with each other, with our environment, and with devices and systems. VR uses advanced display and immersive audio technologies to create an interactive, three-dimensional (3D) environment. AR uses digital technology to overlay virtual objects onto the physical world to provide information and embellish our experiences. Current and envisioned application areas include education, healthcare, professional training, architectural and product design, remote interaction, and entertainment. Continued progress in the burgeoning field of AR/VR requires researchers with Ph.D. training and research experience spanning multiple disciplines including electronic and computing systems, perceptual and cognitive neuroscience, optics and imaging, computer vision, acoustics and audio, and human-computer interfaces. To address this need and realize the transformative potential of AR/VR technologies, this National Science Foundation Research Traineeship (NRT) Award to the University of Rochester will facilitate the development of a structured, multi-disciplinary Ph.D. training program on AR/VR. The project anticipates training 62 PhD students, including 12 funded trainees, from Electrical and Computer Engineering, Optics, Biomedical Engineering, Brain and Cognitive Sciences, Computer Science, and Neuroscience. In addition, the project will benefit approximately 300 other STEM graduate students who will participate in aspects of the training and professional development. Trainees will gain the vision and skills to advance AR/VR technologies as well as an appreciation for the broader cultural and societal implications of these technologies. The project will train inclusive cohorts of scientists and engineers to contribute to society as technical leaders in industry, academia, and government.

The project will train a new cohort of Ph.D. students with a unique set of competencies in the AR/VR domain. It will help shape how future scientists and engineers will be trained not only in AR/VR but more broadly in human-data-system interfaces. The project will advance interdisciplinary research with an innovative theme: integration of quantitative models of human perceptual-cognitive processes into cross-layer design approaches to create and quantitatively evaluate new AR/VR technologies and applications. Research thrusts integrated with the training program in a cross-cutting manner will advance the scientific foundations of AR/VR systems and impact the design of next-generation AR/VR systems. These research thrusts, corresponding to four layers of the AR/VR problem domain, are: (1) AR/VR platforms and computation, (2) perceptual-cognitive aspects of AR/VR design, (3) machine intelligence for AR/VR systems, and (4) AR/VR interfaces and applications. The training program contains three new innovative courses addressing the diverse backgrounds of incoming trainees, exposing them to AR/VR challenges and providing competency to work on AR/VR projects within multi-disciplinary teams as well as a variety of structured professional development activities. In addition, the training will include industry internships and immersive professional development encounters with industry leaders. Both the graduate training model and its outcomes will be widely disseminated to the broader academic community through organized events and a web presence.

The NSF Research Traineeship (NRT) Program is designed to encourage the development and implementation of bold, new potentially transformative models for STEM graduate education training. The program is dedicated to effective training of STEM graduate students in high priority interdisciplinary or convergent research areas through comprehensive traineeship models that are innovative, evidence-based, and aligned with changing workforce and research needs.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This award will support travel and attendance costs for students, postdoctoral fellows, and faculty speakers at the 31st Center for Vision Science (CVS) Symposium on Active Vision. The Symposium will be held at the University of Rochester June 11-14, 2020. Active vision, the study of vision in the context of natural motor behavior, has grown and expanded greatly in recent years, in part due to advances in computational power, display technology, and ability to track body movements.

A rapidly growing body of evidence indicates that motor activity plays a critical role in the computational strategies used for establishing neural representations and achieving task demands. By bringing together a diverse array of viewpoints and creative methodological approaches to the study of the visual system in action, this symposium aims to uncover common principles of visuomotor computation, identify promising new research directions, disseminate knowledge of recent advances in the field, and attract and help form a new generation of researchers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.