Donald I. A. MacLeod - US grants

Treating the visual process as a casual chain, we use measurements of visual phenomena and visual performance to make inferences about how signals are transformed as they flow through the visual system. In particular we propose to exploit a nonlinearity recently demonstrated in the local visual response that precedes the convergence of signals from different photo- receptors, to analyse the characteristics and visual consequences of optical and neural processes that precede this nonlinearity and of the neural processes that come after it. It is possible to monitor the effects of this early local nonlinearity selectively (without intrusion by later nonlinearities, which are pervasive in the visual system), by stimulating the eye with grating patterns too fine to be resolved except at or near the receptoral level, where processing is still strictly local. When such patterns are briefly presented, keeping space-average luminance constant, the spatially modulated stimulus penetrates to and acts upon only those stages that can resolve the strips. But an early nonlinear process, transforming the signal at a stage where resolution is still preserved, can change the space-average excitation of later poorly resolving elements to an extent that depends upon the modulation of the unresolved grating. In many of our experiments we proceed by manipulating, more or less independently, the spatial and temporal characteristics of the stimulus gratings and of the distortion products derived from them. In others, we examine how the behavior of the nonlinear mechanism (as monitored through the visibility of its distortion products) is affected by the contrast of other stimulation. The stimuli used are interference fringes that are formed on the retinal at high contrast without substantial attenuation by the optics of the eye. Difference-frequency gratings are the main type of nonlinear distortion produce that we use for this purpose. Like moire' patterns they occur at a frequency equal to the vector difference in spatial frequency between the two grating stimuli that generate them. The role of nonlinear distortion in contrast sensitivity for a single grating is also to be investigated. These questions are approached both by threshold and by nulling methods.

Color vision in humans depends on particular receptor cells called cones in the retina of the eye. One population of cones, known as the short-wavelength cones or S-cones is most sensitive to light in the blue region of the spectrum; others are the medium-wavelength or M-cones, and long-wavelength or L-cones, most sensitive to red. Cones contribute to two conceptual channels of information leading to visual perception. Intensity of light stimuli is signalled in the luminance channel, and color of light is signalled in the chromatic channel. Sorting out the relative contributions of the different kinds of receptors to these different channels has been an important problem. This project will examine the temporal properties of the chromatic and luminance contributions. A human subject will set flicker of a colored test light to be at just the threshold of visible flickering. The frequency or the intensity of the flicker, or the background, or a second stimulus that tends to mask or augment the flicker of the first, can all be varied. Using very intense lights, this lab already discovered that S- cones can contribute to the luminance channel. Fast responses to light intensity apparently use a brisk pathway to a luminance channel, while slower responses to color use a sluggish pathway to a chromatic channel. The current project will asses the relative prominence and the temporal properties of the two pathways at more normal light levels, and try to determine if the S-cone contribution to luminance is important at these lower levels. This work uses innovative techniques and methodology for delivering and analyzing pulsing stimuli, and results from this work will be very important to theories of color vision, to commercial engineering measurements of colored lights, and to understanding human vision in general.

The main focus of the proposed work is on the question: what limits visual resolution? Treating the vision as a causal chain, measurements of visual phenomena and visual performance are being used to make inferences about how signals are transformed as they flow through the visual system. In particular, we are exploiting a nonlinearity demonstrated in the visual response that precedes the convergence of signals from different photo-receptors, to analyze the characteristics and visual consequences of optical and neural processes that precede this nonlinearity and of the neural processes that come after it. It is possible to monitor the effects of this early local nonlinearity selectively (without intrusion by later nonlinearities, which are pervasive in the visual system) by stimulating the eye with grating patterns too fine to be resolved except at or near the receptoral level, where processing is still strictly local. When such patterns are briefly presented, keeping space-average luminance constant, the spatially modulated stimulus penetrates to act upon only those stages that can resolve the stripes. But an early nonlinear process, transforming the signal at a stage where resolution is still preserved, can change the space-average excitation of later poorly resolving elements to an extent that depends upon the modulation of the unresolved grating. In many of our experiments we proceed by manipulating, more or less independently, the spatial and temporal characteristics of the stimulus gratings and of the distortion products derived from them. In others, we examine how the behavior of the nonlinear mechanism (as monitored by the visibility of its distortion products) is affected by the context of other stimulation. We use laser interferometry to by- pass contrast attenuation by the eye's optics to produce fine stimuli of high retinal contrast, which can be used to characterize the spatial organization of dynamic visual sensitivity regulation to light. Further experiments will reveal the consequences of early visual nonlinearity for the definition of luminance, for color disc rimination and for form perception and motion processing; the first two of which have important implications for users and designers of computer displays. The central nonlinear process of pattern adaptation is also proving useful as an indicator of the quality of the visual representation as it is transmitted from eye to brain, with our recent discovery that patterns too fine for the subject to resolve subjectively can nevertheless penetrate to the cortex and activate pattern-specific neurons there. Proposed experiments exploit this to further advance the analysis of visual resolution losses.

Research on 3D image/video perception in the light of general principles of stereo
processing in the human visual system is being used to derive 3D quality metrics for 3D
video applications in order to deliver the best 3D experience.

Human observers can detect differences in depth with high sensitivity, but limited
precision. Moreover, while the visual system can represent fine details of the 2D image
that are carried by high spatial frequency components (even when the image is rapidly
changing), it can not track variations in depth with comparably high resolution in space
or time. Thus the representation of stereoscopic depth is restricted both in bandwidth and
in bit depth. Because of those limitations, some deviations from accuracy in the
representation of depth at the retinal level are perceptually salient, and others less so.
Measurements of perceived image fidelity across a range of spatial and temporal profiles
for the depth signal are being used to guide the development of optimal video processing
techniques, and to allow evaluation of the advantages and limitations of alternative 3D
video coding algorithms such as multiview versus video+depth).

Vision experiments investigate both perceived fidelity and perceived image quality in 3D
video generated using a variety of encoding schemes. From those results quality metrics
are developed and integrated into video processing and communications applications. A
human-centric disparity estimation and view synthesis algorithm is being developed for
video processing and communications applications; this can also be used to improve the
performance of object detection, classification and tracking, and to generate multi views
for autostereoscopic display, which finds applications in 3D enabled diagnostic medical
imaging and surgical systems.