Fulvio Domini - US grants

In our daily life, we move in the environment, grab objects and perform a number of actions that are fundamental to our survival. These actions may seem very natural and effortless despite the fact that the brain performs an extremely complex analysis of the light pattern that falls on our eyes in order to determine the structure and shape of the surrounding objects. This problem is very difficult to solve since objects are three-dimensional but our eyes only register their two-dimensional projection (also called retinal image), like the film in a camera.

Most vision scientists have approached this problem by asking the following question: How does the brain derive the 3D structure of objects from the information that is present in a certain instant of time in a certain region of the retinal image? Because the image on the retina is two-dimensional and time can be added as a third dimension, the visual stimulation can be represented in a three-dimensional space. The research conducted so far has focused on the problem of how local regions of this space-time domain are analyzed by the visual system, while the problem of how the visual system is capable of integrating the information contained in different spatial-temporal regions has been neglected.

The overall objective of the present research project is to investigate the spatial-temporal integration of information in the recovery of 3D shape from retinal projections. In particular, three goals will be pursued. First, the research will investigate in which manner local visual processing is affected by interactions with stimulus information present in different space-time locations. Second, the research will exploit the stimulus conditions that are responsible for spatial and temporal organization. Third, the research will determine whether spatial and temporal interactions occur among different sources of depth information. Understanding how the human visual system solves this problem will not only be a valuable advance in the study of visual perception but could also produce novel insights toward the building of machines that mimic our behavior and interactions.

Long ago artists discovered how cues like linear perspective, shading, cast-shadows, and occlusions create the illusion of a three-dimensional world on a two-dimensional canvas. Likewise, vision scientists have long assumed human perception exploits similar depth-cues to construct a three-dimensional experience of visual objects. It has been postulated that depth perception is informed by separate and different depth-cues that may effectively add up to a unified percept. However, this conventional wisdom may fall short of the mark.
With NSF support Dr. Fulvio Domini will test whether depth cues are intimately interrelated, whether perception is based on the relation among depth cues not cues considered separately. This bold hypothesis will be developed as a model and tested in experiments with human participants. Broader Impacts include implications of the basic research for studies of visual impairments in disease and aging, and for the development of robust computer vision algorithms.

Human observers perceive a three-dimensional world even though images on the retina are two-dimensional. How this can be done has been a classic problem in visual perception. Researchers have isolated a list of "depth cues", such as binocular disparity and retinal velocity, that might inform the visual system about the third dimension. Several models have been developed to explain how perception of the third dimension depends on each cue in isolation. However, what remains to be explained is how the visual system operates when several depth cues are simultaneously available in the retinal projections.

With support of the National Science Foundation, Dr. Domini will test a new depth-cue processing model that differs from the current cue-combination model in two respects: (i) it computes relative but not absolute estimates of the third dimension of depth and (ii) rather than being formulated in terms of averaging the depth magnitudes recovered from each cue in isolation, its estimates are based on sophisticated statistical analyses of the multidimensional cue space. Experiments with human observers will determine whether the model accurately captures the phenomena of human visual perception, including both the successes and errors that that human vision shows. In addition to advancing our basic understanding of visual perception, the results of this research may aid the development of more effective computer vision principles that could be used in the design of robots.

When we perform mundane daily actions like picking up a cup of coffee, our brain needs to figure out the location of the cup, its shape and weight (which changes depending on how full it is). We tend to think that our perceptual experience of the cup is what determines our interaction with it. However, several studies over the past two decades have repeatedly shown that a perceptual task, like judging the size of an object or its weight, is processed by a different part of the brain than an action task, like reaching to lift the object. This project takes an alternate view, in which the brain processes the visual scene, but this process may be subject to errors (like overestimating the size of a cup or its weight). These errors are immediately detected while an action is unfolding and the subsequent movements toward the object are quickly corrected. This research plan will explore the nature of these complex corrections. It will also examine certain circumstances in which our perception is faulty while our actions are accurate. This knowledge could help people rapidly learn new visuomotor skills, such as interacting in virtual-reality environments and teleoperation. Indeed, a more comprehensive understanding of visually guided action could inform the development of these emerging technologies. Finally, discoveries from this proposal could help improve the lives of individuals with neurological disorders, which often lead to a profound loss of motor ability that significantly impairs activities of daily living.

This research project uses state-of-the-art virtual reality environments to test 1) to what extent humans can adjust their motor actions "on the fly" to compensate for inaccuracies in visual perception and 2) whether visual perception changes when smooth movement coordination cannot be achieved through motor adjustments. Three mechanistic hypotheses will be tested in which sensory-prediction errors - signals produced when sensory feedback does not match one's expectations - enhance the accuracy of action during repeated visuomotor interactions. At the core of each hypothesis is the idea that when biases in perception lead to inaccurate movements, sensory-prediction errors will drive adaptive changes across the sensorimotor system. Three non-mutually exclusive mechanisms of adaptive change will be tested: (1) Rapid re-alignment of the motor output with the physical world, (2) Changes in calibration of visual perception, and/or (3) Selective changes in the contribution of specific aspects of visual information to action. To test these hypotheses, an integrated set of behavioral experiments will be conducted, in tandem with the development of computational models to mechanistically explain the closely intertwined roles of perception and action.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Despite at least a century of scientific investigation, it is still not entirely understood how the human brain constructs a perception of three-dimensional (3D) objects and space from the 2D information reaching the eye through light rays.  It has been known for a long time that this ability is attained on the basis of a variety of different visual signals (that the brain interprets) called “depth cues”, which the brain then combines together to derive the 3D structure of the scene.  Examples of depth cues can be observed in paintings where linear perspective, shading, and even simple contours depict a 3D world on the flat canvas just as they do on the flat human retinas while people observe real 3D objects. However, how these cues encode 3D information and how the different cues are combined to generate our stable coherent perception of 3D visual space and objects remains poorly understood.  The prevailing theory in the scientific literature (Bayesian probabilistic inference theory) postulates that the brain derives a 3D structure of the world by determining how likely a particular 3D structure is given the information on the retina. Implementing this model requires a host of assumptions, such as that depth cues deliver “noisy” estimates of 3D parameters that are still, on average, accurate. However, a number of common and important observations cannot be fully explained by the Bayesian model and cast doubt on the critical computational assumptions of the model. Moreover, the Bayesian model struggles to explain important differences in the “quality” of 3-dimensionality that we perceive between viewing the real world and artificial situations such as pictorial images, or virtual or augmented reality (VR, AR). This project tests a new theory (the Intrinsic Constrained theory) that makes an entirely different and simpler set of assumptions compared to the Bayesian theory, but that can predict a wider range of perceptual phenomena.  The project integrates the IC theory with a recently proposed theory that postulates that the visual system does not generate a single encoding of 3D space, but two distinct encodings, one which is relevant to understanding the scene (i.e., 3D object shape and layout) and the other that underlies visually guided movements like reaching for and grasping an object.  The latter encoding is claimed to underlie the special subjective experience of 3-dimensionality that is most obvious while viewing stereoscopic images (e.g., 3D movies). This project will show that the IC model can efficiently incorporate the claims of two distinct representation of 3D space. In doing so, it is able to provide a better explanation of a range of fundamental aspects of our perception of 3-dimensionality that are challenging to explain with the prevailing model, including those required for a better understanding of factors important for developing 3D technology (e.g., VR and AR).

This award supports empirical research that tests a computational model arising from the Intrinsic Constraint theory of cue integration against the prevailing computational model belonging to the Bayesian framework. Specifically, the new model challenges three main assumptions of the Bayesian model that will be tested by experiments in distinct work packages: (1) that depth cues on average provide veridical (accurate) 3D estimates; (2) that these estimates are stochastic and that their probability distributions are encoded by the visual system; and (3) that the process of cue integration results in a single encoding of 3D structure. Instead, the Intrinsic Constraint model predicts that cue estimates are biased, deterministic, and that cue integration results in two distinct encodings of 3D structure. Using state-of-the-art visual display and motion tracking apparatus that allow both psychophysical and psychomotor response measurements, the investigators are conducting a comprehensive set of experiments aimed at critically testing two theories (Bayesian and IC). The first work package establishes which theory better explains 3D perception based on single or combined depth cues. The second work package establishes if depth cues provide stochastic estimates as proposed by the Bayesian theory or if cues provide deterministic noise as proposed by the IC model, with noise in 3D estimates due to extraneous experimental factors. The third work package shows how differences in the subjective experience of 3-dimensionality is linked to the efficacy of the 3D encoding underlying guidance of movement, but not that underlying perceptual judgements.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.