Michael S. Landy - US grants

Software will be developed that provides experimental control of stimulus display and data acquisition in neurophysiological and psychophysical investigations of visual processing. This software is being developed for three main reasons: (1) to take advantage of advances in computer technology that facilitate the development of such systems, (2) to incorporate an extensive library of low- level visual display drivers that have been developed at NYU over the past five years, and (3) to incorporate modern design concepts into the software. The software is expected to provide the basic experimental control program for the next 5-10 years for the research group involved in this proposal. In addition, a key design feature of the software system is portability, with the goal of freeing its general structure from dependance on a specific processor architecture or on a particular set of input and output devices. The common theme of the research that will initially use the newly developed software is that it relies upon computer controlled generation and display of visual patterns, either on monochrome or color monitors along with the collection of stimulus-evoked behavioral and neural responses. Although the initial effort will concentrate on the generation of software for visual experimentation, the design of the system is by no means limited to this area, and adaptation of the general control structures to a wide range of neural and behavioral experiments should be straightforward.

DESCRIPTION: The proposed experiments are intended to clarify how humans make use of the various different cues to depth. Three themes guide this work. The first is that an ideal observer is sensitive to the quality of information available from different depth cues in a particular scene. The second is that the ideal observer is sensitive also to the logical type of information available from different depth cues. Depth information of different types must be promoted to a common type before being combined, and this process may explain many depth cue interactions. The third theme concerns the veridicality of stimuli. If an ideal observer is sensitive to the quality of information available from different depth cues then the use of impoverished stimuli can alter the nature of the processing. These three themes lead to a Modified Weak Fusion framework for understanding depth cue combination. The experiments will use highly realistic, controllable viewing conditions involving large displays and accurately modelled scene and surface properties. The proposed research consists of three major tasks: 1) Examination of interactions between depth cues in terms of perceived depth as well as other perceived surface properties such as lightness to test the MWF model 2) Carry out a number of single cue depth studies and other control studies and 3) Study the form of the surface representation used by human observers.

DESCRIPTION (provided by applicant): Biological visual systems make use of many different sources of information ("cues") for visual judgments. For depth and shape estimation, for example, these include occlusion, texture, perspective, motion parallax, disparity, shading and contour. The combination of these cues is based on the relative reliabilities of the individual cues, but cannot occur until cues are promoted to a commensurate scale by filling in one or more needed parameters (e.g., the fixation distance and azimuth for depth and slant estimates). These parameters are also estimated using multiple cues (e.g., both retinal and oculomotor cues for the viewing geometry). We propose statistical decision theoretic models for ideal behavior in the visual estimation of scene properties and for movement planning. The ideal observer or actor must take into account measurement uncertainty, associated with different outcomes, and prior information about the current state of the world. We propose experiments intended to clarify how human observers promote and combine cues for vision and for the visual control of action. The experimental methods used are based on perturbation analysis which permits examination of a system that can potentially react to distortions and inconsistencies in the stimuli. The proposed research consists of three major tasks. (1) We will analyze observer behavior relative to predictions of ideal Bayesian decision makers confronted by the same levels of uncertainty in tasks of perceptual decision, reaching and grasping. (2) We will examine cue combination in the service of cue promotion, again with reference to ideal behavior. (3) We will continue our studies of spatial interpolation performance so as to better understand such aspects of the underlying model as the prior distribution, and the methods used by the observer to be statistically robust (which, in this context, is closely related to the scene segmentation problem).

A fundamental question in perceptual and cognitive science concerns how decisions about incoming sensory information unfold over time. The influx of available sensory information must be balanced across time with the need for quickness versus accuracy in making a decision. Sensory systems are flexible in response to changing context, resulting in several dynamic aspects of perception at multiple time scales, including the trade-off between speed and accuracy in decisions or actions, adaptation for repeated sensory stimuli, and long-term recalibration in response to a consistent change in stimulation or error. This proposal attempts to advance theoretical understanding of these processes by unifying disparate threads in modern perceptual theory while developing models of the dynamics of sensory integration and decision-making. The research could ultimately have implications for the design of virtual reality systems, lighting and sound systems, visual displays, and artificial vision and sound processing systems.

A large literature on multiple cue integration, or how observers integrate multiple sources of information for making a perceptual decision, suggests that human behavior compares favorably with the predictions of optimal Bayesian models for combining multiple sources of information (including prior knowledge). However, this literature examines decisions under static conditions. There is a mostly distinct literature on speeded decision-making that instead asks how information is accumulated over time, leading to the combined decision of both what response to make and when to make that response. The novelty of the proposed research is to unify these two threads of perceptual theory by developing and testing a model that incorporates both a dynamic decision-making model for evidence accumulation (a diffusion process) and multiple-cue integration. Each cue contributes independently to the dynamic process of evidence accumulation. The experimental work proposed to test the model covers a wide range of sensory decision-making phenomena and thus is expected to have a strong impact on the field of perceptual science. However, the wider impact will result from the unifying models to be developed, as the combination of optimal cue integration with dynamic decision-making models has extremely wide applicability in cognitive science generally. A companion project is being funded by the French National Research Agency (ANR).

Project Summary To optimally estimate a property of the environment such as object size, location or orientation, one should use all available sensory information and combine it with prior information, i.e., a probability distribution across possible world states, reflecting knowledge of scenes one is likely to encounter. Sensory input typically arises from multiple sensory modalities, and is uncertain due to physical and neural noise. How are these sources of information combined? An ideal observer will combine all sources of information, taking into account the relia- bility of each source. In addition, such an observer needs to consider alternative causes of discrepancies be- tween sources of sensory information. Do two sources disagree so much that one should conclude they derive from different objects, and therefore have separate causes in the environment? Or, does a discrepancy indi- cate that one or both sources of information (e.g., sense modalities) have become uncalibrated? Many studies define ?optimal? cue integration as maximizing the reliability of the combined-cue estimate, which is generally consistent with human behavior. Do observers have access to the resulting reliability estimate to determine one's confidence in this estimate, perhaps to inform subsequent behavior? What computation does the brain use to solve these problems and how are these computations implemented? We propose research aimed to answer these questions. In our first aim we propose to develop biologically realistic models of how such computations are implemented, i.e., testable neural-network models of optimal behavior for sensory estimation, causal inference, recalibration and confidence. Second, we propose a series of experiments in an area that has been little studied in the framework of optimal cue integration: the combina- tion of visual, tactile and proprioceptive inputs for localization. These experiments test whether humans per- form optimal integration and recalibration of multisensory cues and priors under unclear causal structures in scenarios that are more complex than typically studied (i.e., involving dynamics, context effects, etc.) and thus more similar to the real world. These studies are important and innovative on their own. In addition, they will also provide the foundation for Aim 3, in which we will probe the implementation of cue combination, influence of priors, causal inference, recalibration and confidence in the human brain using fMRI. Together, the experi- mental data from Aims 2 & 3 will be used to test the models from Aim 1. These studies will shed light on the way in which multisensory stimuli are encoded to form a coherent percept, the information considered when perceptual decisions are made, and how vision is used to guide us in an ever-changing world. These experi- ments on normal humans will provide a starting point for understanding multisensory perception and perceptu- al adaptation in individuals in which these systems are compromised by conditions that impact sensory input (e.g., amblyopia, AMD, stroke).