Martin Banks - US grants

The research in this proposal focuses on the development of form perception during early infancy. The proposal consists of two different yet related lines of research. The first involves the study of human infant form perception and its development. The experiments explore basic mechanisms using the contrast sensitivity function. They also involve investigating basic mechanisms of form perception such as feature detectors and contrast constancy. The two visual systems theory as applied to early perception will be tested. Finally, mechanisms of local and global analysis will be examined developmentally. The second line of research involves the development of some sensorimotor abilities required for acute form perception. It is argued that the growth of refraction and accommodation is intimately involved in the development of form perception. Several experiments are proposed including techniques for infant assessment, integration of sensorimotor abilities, and measurements of image quality. It is argued that the development of accommodation provides an excellent model for studying principles of general sensorimotor development.

Ten visual scientists request instrumentation for image processing. The requested equipment consists of five major components, located in two adjacent rooms: (1) a high- performance, interactive color display system and monitor, (2) a host computer with network connection, (3) video cameras for digitizing, (4) an editing videocassette recorder for inputing and outputing image data, and (5) a high-resolution monitor suited for 35mm photography. In addition, the investigators are requesting a file server computer, located in the Campus Computation Center, that would provide mass storage and serve as the central node in an Ethernet network of workstations in the various laboratories and the image processing station. An extremely versatile and powerful image processing system is needed to meet the diverse needs of the participating investigators. The display system must be able to accept images from clinical sources (e.g., videotaped fundus images), histological tissue preparations (e.g., autoradiographs), and natural scenes. It must have good spatial resolution to enable accurate localization and measurements on features in the digitized images. It must also be able to perform a variety of transformations rapidly on the digitized images. In addition, several of the investigators require the ability to generate a wide range of 2-dimensional, static or time-varying, monochromatic or chromatic stimuli for use in psychophysical and electrophysiological experiments. The requested equipment suits the diverse needs of the group of participants very well, and would enhance the research programs of everyone involved.

Many aspects of spatial vision improve significantly during the first year of life At birth, contrast sensitivity, contrast discrimination threshold, grating acuity, and vernier acuity are all at least an order of magnitude poorer than mature values. Two primary goals in studies of visual development are: (a) to determine the consequences of such low sensitivity to the perception of various visual attributes and (b) to understand the optical and neural underpinnings of perinatal deficits and post- natal improvements. This application outlines a program of research to examine several aspects of spatial and chromatic visual sensitivity early in life. The guiding theme for much of the proposed research can be stated as a question: To what extent can age-related changes in pre-neural structures (e.g., optics, and photoreceptors) account for early spatial and chromatic visual deficits and their subsequent improvement? Answers to this question will help delineate the relative contributions of pre- neural and neural development to age changes in visual function. There are four specific aims. (1) We will construct ideal observers with the characteristics of the eyes and photoreceptors of infants of various ages. Their performance will be examined for a variety of spatial and chromatic visual tasks. Disparities in ideal performance at various ages will help delineate the consequences of immature optics and photoreceptors. (2) We will examine several aspects of the development of luminance contrast sensitivity and grating and vernier acuity. The experiments will provide useful data in their own right, but they are particularly important to deducing the consequences of immature photoreception on the development of performance in these tasks. (3) We will examine the development of chromatic sensitivity, too. Again these experiments will provide useful data in their own right, but they are designed specifically to determine if age changes in overall visual efficiency are sufficient to explain the development of chromatic discrimination or whether age changes in chromatic mechanisms per se are implied. (4) Finally, we will study the development of spatial phase discrimination using a paradigm that has illuminated fascinating anomalies in normal adult peripheral vision and in the central vision of adult amblyopes. We will examine the relationship between these phase anomalies and infants' phase sensitivity.

9309820 BANKS Humans are able to move rapidly through complex environments while avoiding stationary and moving obstacles; clearly the sensory modality most important for this skill is vision. The ease with which people use visual motion information to guide navigation belies the underlying complexity of the task. Self-motion through an environment produces a pattern of movement on the retina called the optic flow field. An influential early proposal was to identify the direction of self-motion with respect to obstacles by locating the source of flow, i.e., the focus of expansion, but the task is actually much more complicated. First, since the sensing of 2D motion (which is required to derive the optic flow field in the first place) is a difficult problem in its own right, one cannot assume that a noise-free vector field is available for calculation of observer motion. Second, since people commonly move their eyes and head while locomoting, these movements obliterate the focus of expansion. Third, since visual scenes often contain moving objects besides the observer, those objects do not produce a consistent focus. Despite these complications, people use the optic flow field very effectively to judge heading relative to landmarks and other moving objects. This research is mainly concerned with how efficiently human observers use the information contained in the optic flow field to determine the parameters of their self-motion. A measure of efficiency derives from the comparison of the performance of an ideal observer for heading tasks to that of human observers in the same tasks. Because the ideal observer uses all of the information in the flow field, its performance provides a rigorous benchmark against which to compare human performance. Specific comparisons will not only allow the measurement of human efficiency but also the identification of some of the causes of inefficiency. The research will demonstrate how variables such as number of elemen ts in the display, type of flow field displayed, sharpness and size of the display, position of stimulation on the retina, presence of rotational flow due to eye/head movements, and knowledge of the scene geometry affect human efficiency in determining the direction of self-motion. While the products of this research will enhance our understanding of space and motion perception, they may also have important practical consequences. For one thing, since biological systems have evolved robust mechanisms to subserve visually-guided navigation, a better understanding of how this is accomplished should lead to better algorithms for mobile robotic systems. Second, the development of an ideal observer for heading tasks will provide a benchmark against which to compare the performance of computer algorithms. Third, since perception of heading with respect to stationary and moving objects is crucial for driving and flying, a better understanding could lead to improved procedures for screening and training drivers and pilots and for designing instruments, cockpits, and roadway and runway markings. ***

DESCRIPTION (provided by applicant): A fundamental problem faced by the visual system is providing information about the 3d environment from the 2d retinal images. Perhaps the most precise source of information arises from the fact that the two eyes have different vantage points. This means that images on the two retinae are not identical. The differences between the locations of matching features on the retinae are binocular disparities and the ability to perceive depth from these disparities is stereopsis. Investigations of inferring 3d layout from disparity fall into two general categories: 1) the estimation of disparity from the retinal images and 2) the interpretation of the estimated disparity. Specific Aim 1 concerns disparity estimation and Specific Aims 2 and 3 concern disparity interpretation. In the experiments and modeling associated with Specific Aim 1, we will examine the spatial and chromatic properties of disparity-estimating mechanisms. We will, for example, determine whether the highest stereo resolution, the disparity-gradient limit, and the difference in stereo sensitivity with luminance as opposed to chromatic stimuli result from using a binocular-matching algorithm that provides piecewise-frontal estimates of the depth map. In some of these experiments, we will improve the optics of the eye and investigate the costs and benefits to stereovision. In the experiments and modeling associated with Specific Aim 2, we will investigate whether disparity and texture slants signals are combined in a weighted sum (for estimating slant) and a difference (for assessing texture homogeneity). We will also examine how changes in the reliability of disparity and texture signals affect these two processes. In the experiments associated with Specific Aim 3, we will study a widely experienced perceptual phenomenon: adaptation to the 3d distortions that result from horizontal magnification of one eye's image. We will try to pinpoint the adaptation mechanism and to determine whether subjects can adapt to two states simultaneously. We will also examine adaptation to a vertical magnification of one's eye image.

PI: Banks, Martin

The proposed research will investigate the means by which we see 3-dimensionally via binocular vision. The research will focus on two issues: 1) how the human visual system solves the "matching problem" in binocular vision and 2) how the visual system represents surface shape and orientation from binocular depth cues.

The matching problem in binocular vision has been actively researched for decades. We still do not fully understand how the problem is solved. The matching problem is simply stated in the following way: For every image point in the left eye, the visual system must find the appropriate point in the other eye to match with it. With N possible points, the number of theoretically possible matches is N4 so with large N the matching process can become computationally unmanageable. The visual system can massively reduce the number of possible matches by using what is termed the epipolar constraint. For an image point in one eye, its match must lie on the corresponding epipolar line in the other eye. By using the epipolar constraint, the matching problem can be reduced to a one-dimensional search. However, when the eyes' positions change (e.g., fixating from far to near), the positions and orientations of corresponding epipolar lines change on the retina. Thus, to implement the epipolar constraint, the visual system must take the eyes' positions into account. We have developed an experimental procedure that will allow us to determine whether the visual system uses the epipolar constraint and, if so, what signals the system uses in order to calculate how epipolar lines ought to move when the eyes move.

We will also examine the means by which surface shape is represented in the visual system. An important cue to surface shape is the pattern of horizontal disparities arriving at the two eyes, but those disparities by themselves cannot yield a veridical estimate of shape. Other signals such as vertical disparities or eye-position signals must be used as well. We have shown in the previous grant period that changes in the eyes' vergence can cause a compelling change in perceived shape even when the retinal disparities are completely constant. We also know that prolonged viewing of a curved surface causes a subsequently viewed flat surface to appear curved in the opposite direction. Aftereffects like this have been called "disparity aftereffects" because the explanations offered refer to disparity encoding alone. In the proposed research, we will examine the curvature aftereffect. By manipulating eye-position signals and retinal disparities independently, we can determine whether the aftereffect is caused by adaptation among disparity-encoding mechanisms (which is the prevailing theory) or whether it is caused by adaptation in higher-level, shape-encoding mechanisms. Preliminary measurements suggest that the latter hypothesis is a better predictor of the data. We will also determine how the various signals involved (e.g., eye-muscle signals and vertical disparities) are weighted under different viewing conditions.

The proposed work will yield a better understanding of these aspects of binocular vision and, consequently, may yield insights into improvements in visual aids such as binocular microscopes, head- and helmet-mounted displays, and other realistic displays.

ABSTRACT NOT AVAILABLE

DESCRIPTION (provided by applicant): Three-dimensional computer graphics plays a key role in basic research on visual space perception and in numerous applications such as virtual reality. The goal in 3d graphics has been to create the display on a digital display screen that yields the same 2d retinal images as the real 3d scene itself. This can be done stereoscopically by presenting two images on the display screen, one for each eye. The resulting 3d impression can be very compelling. However, the resulting percept is frequently different from that obtained with real scenes and objects. Part, if not all, of the problem can be attributed to inappropriate screen cues that signal the distance to (and shape of' the display screen rather than simulated scene or object. Here we propose to measure the contributions of those screen cues and to develop a novel, multi-focal graphics display system that will circumvent the most problematic of the screen cues. The research effort requires the combined talents of psychophysicists, optical scientists, and computer graphic engineers, so we are requesting funds to put together this collaborative research program. There are four specific aims.1. Measure the influence of digital display cues on perceived 3-dimensionality. We will investigate the contributions of three such cues: a) inappropriate motion parallax during head movements, b) pixelization, and c) inappropriate focus cues.2. Develop an adaptive lens. The requirements are a) high image quality, b) ability to change power by 2 diopters or more, c) frequency response of 180Hz or higher, d) synchronizing lens power changes with the graphic display, and e) small and light enough to be worn by a human observer.3. Develop and evaluate the graphic display algorithm required to simulate focus cues accurately.4. Construct and evaluate the lens and graphical display algorithm. We will test human observers with and without the system in use. We will assess the quality of the resultant in terms of apparent depth, flicker, sharpness, smoothness of motion, and more. We will be particularly interested in using depth-judgment tasks to evaluate the effectiveness of the system in producing more veridical 3d percepts.

One reason why pictures are such powerful forms of communication is that they allow viewers to perceive three-dimensional information on a two-dimensional surface. At one level, it may seem obvious how this feat of dimensional expansion is accomplished: a 2D picture generates an image on the retina that is similar to the image that would be generated by the full 3D scene. However, this is strictly true only when a picture is viewed from the geometrically correct position--the center of projection. When pictures are viewed from angles, the resulting retinal image is quite distorted relative to the center of projection, yet the viewer is still reconstruct the 3D scene. Photographers, painters, computer scientists, cinematographers, and vision scientists have all wondered how this works; that is, how are people able to perceive scenes in pictures correctly across a variety of viewing positions?

With support of the National Science Foundation, Dr. Banks will investigate the means by which viewers perceive scenes in pictures across various viewing positions. He and his research team will examine various viewing conditions including stereoscopic pictures and wide-angle pictures in order to determine those conditions in which people are most able to perceive picture content accurately. In addition to advancing our basic understanding of visual perception, this work may lead to the development of new techniques for presenting pictorial information.

DESCRIPTION (provided by applicant): Our subjective impression of the visual world is that everything is in focus. This impression is reinforced by the common practice in photography and cinematography of creating images that are in focus everywhere. Our subjective impression, however, is quite incorrect because most parts of the retinal image are significantly blurred. We will examine the use of blur and the eye's focusing response in the perception of distance and size. We will also examine how blur and accommodation affect viewer fatigue. There is clear evidence that blur affects distance and size perception. A probabilistic model will be developed that incorporates the information in blur and other cues. The model will be evaluated and refined by comparing its behavior to human perception in a series of psychophysical experiments. One set of experiments will use miniaturization effects developed in photography. Those experiments will determine the contributions of blur and the eye's focusing response (accommodation) on perceived absolute distance. Another set of experiments will capitalize on the information contained in the blur of a border between two regions in an image. Those experiments will determine whether the plausibility of observed blur, focus distance, and relative size affect the perception of depth order. The geometries of depth from binocular disparity and depth from blur are fundamentally similar. In principle, the two cues provide complimentary information about distance, with disparity providing more precise information near the point of fixation and blur providing more precise information away from fixation. The relative contributions of disparity and blur to perceived distance and eye-movement control will be measured for various positions relative to where the viewer is looking. The amount of blur in the retinal image depends on the absolute and relative distance of objects in the environment and on where the viewer is focused. We define the concept of natural depth of field; this is the appropriate relationship between other depth cues and blur for a normal human eye viewing the natural environment. We will determine whether people are sensitive to natural depth of field and whether 3D structure is misperceived in images that deviate from the natural relationship. Stereo displays are becoming more widely used in applications such medical imaging, surgical training, and scientific visualization. The motion picture and television industries are also rapidly introducing stereo. With stereo displays, the normal correlation between vergence and accommodation is disrupted. The result is a vergence-accommodation conflict. There are several adverse consequences of the conflict, but the most important is undoubtedly discomfort and fatigue. We will determine how viewing distance, refractive error, age, and the magnitude and sign of the vergence-accommodation conflict affect discomfort and fatigue. A better understanding will aid the design of stereo displays and the content shown on such displays.

How the visual properties of an object or scene affect human visual perception is a fundamental question in cognitive science. The proposed research is concerned with the visual properties of surfaces, which can reveal many characteristics important to decision-making, such as the identity of a substance, the cleanliness of an item, or the authenticity of a photograph. Humans make these judgments by analyzing the light reflected from surface materials. The proposed research examines how this is accomplished. Shiny materials like polished metal, gloss paint, smooth plastic, and varnished wood yield bright reflections of the light sources that illuminate them. Humans are quite adept at distinguishing those reflections from other causes of bright spots, such as white matte paint. Reflections are view dependent, meaning that their apparent position on the surface changes depending on the location of the viewing eye. Because of this view dependence, the reflection usually appears to lie beneath the surface. The proposed research will examine how the apparent position of reflections affects human judgments of material. The investigators will conduct theoretical work on the properties of reflections from many types of materials and experimental work on how well humans are able to use these properties to identify material. The investigators will also develop new computer graphics algorithms and a new display technique to allow realistic presentation of reflections from realistic materials.

The work will contribute significantly to basic science and to several applications. One important application concerns how industry describes the visual properties of materials. Currently, manufacturers of paint, plastics, and furniture use one-dimensional scales of appearance, such as gloss, and inexpensive appearance measurement devices, such as glossmeters, to categorize finishes by "gloss level." Very different materials can have the same gloss level even though they produce very different reflections. The investigators will determine whether materials with the same gloss level actually look the same. The results could affect the way industry measures and assesses material. Another important application concerns how people distinguish between computer-generated and real images. This has important implications, not only for the motion picture and computer graphics industries, but also for forensic sciences. The principal investigator, in collaboration with an expert in digital forensics, will explore whether the pattern of light dispersal from human skin in computer-generated and real people, and the associated parallax properties, can aid in distinguishing fake from real images.

? DESCRIPTION (provided by applicant): Amblyopia is a deficit that arises from abnormal visual experience early in life, most commonly when the two eyes are not aligned (strabismus) or have unequal refractive error (anisometropia). It has long been thought to develop into a permanent deficit unless properly treated early in life; however, recent studies call this into question [1-8]. Adult amblyopes can recover some visual functions. Thus, amblyopia provides a useful model for understanding how to unlock adult neural plasticity. Our main aim is to develop new interventions for restoring visual acuity and stereopsis in adults with amblyopia by embedding visual training in active and rewarding visuomotor tasks that require stereopsis. Our work during the current grant period indicates that playing action-packed video games enhances several aspects of vision in amblyopic adults. Training either monocularly or under dichoptic conditions results in improved visual acuity and reduced suppression in adults with amblyopia. However, it is now clear that both perceptual learning (PL) and videogame play (VGP), whether monocular or dichoptic, result in only a modest improvement in visual acuity (by one to two lines), and only limited improvement in stereopsis [1,3-8]. Much recent work has focused on the role of binocular suppression, as the key to recovering visual functions in amblyopia. Specifically, it has been suggested that treatment should be dichoptic - and aimed at eliminating suppression - rather than monocular [5-8]. However, our pilot studies call this approach into question, and suggest a more direct approach to improving both visual acuity and stereopsis: training under 3D conditions. By forcing the two eyes to coordinate and integrate their signal in the service of a common output, stereopsis provides a unique common teaching signal to the two eyes that may be central to overcoming amblyopia. Our emphasis on stereopsis is novel, but it is based on: (1) our pilot data, showing that in adults with amblyopia training stereopsis directly resulted not only in improved stereopsis, but also in improved visual acuity and contrast sensitivity, as well as reduced inter-ocular suppression and (2) the loss of stereopsis has demonstrable negative effects on everyday activities that can significantly impact individual's quality of life as well as limit their career choices and activities. The most noticeble qualitative deficit associated with amblyopia is impaired or complete loss of stereoscopic depth perception. Thus, the Aims of this proposal are: 1) To develop an integrated, immersive 3D videogame training program for adults with amblyopia; 2) Develop a battery of outcome measures to evaluate the effectiveness, mechanisms and real world impact of training 3) Evaluate the effectiveness of direct training of stereopsis on vision recovery in adult amblyopes.

When the eye is out of focus, the lens inside the eye changes shape to bring the image back into sharp focus. Scientists do not know how the eye knows which direction--in or out--to focus, or how the eye knows when the best possible focus has been achieved. The first aim of this research is to use an innovative adaptive-optics apparatus to manipulate three possible cues that are used to guide the eyes when focusing and to measure the corresponding effects on the eye's focusing response. From the results, the researchers can construct a general model of focusing in human eyes. The second aim of this research is to develop a novel graphics-rendering technique that takes into account the optics of the viewer and creates more realistic, natural images on the retina for virtual and augmented reality (VR and AR) displays. A better understanding of the eye's focusing response and the development of appropriate graphics techniques could have significant practical and clinical benefit. For instance, VR and AR devices, which are rapidly entering the marketplace, are known to cause visual discomfort, decreased performance, and distorted depth percepts. A better understanding of how the eye focuses and determines the three-dimensionality of the viewed scene will help scientists and engineers determine how best to design and evaluate this next generation of displays. The research may also aid the design of clinical procedures such as LASIK eye surgery and accommodating intra-ocular lenses (AOLs). These procedures affect the focusing cues under investigation, so a better understanding of how the cues are used may lead to improvements in these procedures.

The research investigates how the visual system determines the direction and magnitude of accommodative responses needed to assure a sharply focused image on the retina. There are three cues that the eye could use: 1) Chromatic aberration. Some colors, such as blue, are generally focused in front of the retina while other colors, such as red, are usually focused behind the retina. This can help tell whether focus is too near or too far; 2) Higher-order aberrations. When focus is too near, some properties of the retinal image are different compared to when focus is too far; and 3) Micro-fluctuations. The eye's focus state fluctuates at a rate of about 1 to 2 times per second and this could provide information about whether focus is too near, too far, or just right. Previous research has shown convincingly that chromatic aberration is used. Some research suggests that higher-order aberrations are used, but the evidence is not convincing. No research has tested whether micro-fluctuations are used. The present research considers all three cues to construct a cue-combination model of accommodation.

See abstract in Overall Component.

This project seeks to develop head-worn Augmented Reality (AR) systems that look and feel like ordinary prescription eyeglasses, and can be worn comfortably all day, with a field of view that matches the wide field of view of today's eyewear. Such future AR glasses will enable vast new capabilities for individuals and groups, integrating computer assistance as 3D enhancements within the user’s surroundings. For example, wearing such AR glasses, an individual will see around them remote individuals as naturally as they now see and interact with nearby real individuals. Virtual personal assistants such as Alexa and Siri may become 3D-embodied within these AR glasses and situationally aware, guiding the wearer around a new airport, or coaching the user in customized physical exercise. This project aims to advance two crucial, synergistic parts of such AR glasses: 1) the see-through display itself and 2) the 3D eye-tracking subsystem. The see-through display needs to be both very compact and have a wide field of view. To achieve these display requirements, the project uses true holographic image generation, and improves the algorithms that generate these holograms by a) concentrating higher image quality in the direction and distance of the user's current gaze, and b) algorithmically steering the "eye box" (the precise location where the eye needs to be to observe the image) to the current location of the eye's pupil opening. In current holographic displays, this viewing eye box is typically less than 1 cubic millimeter, far too small for a practical head-worn system. Therefore, a practical system may need both a precise eye tracking system that locates the pupil opening and a display system that algorithmically steers the holographic image to be viewable at that precise location. The 3D eye tracking system also seeks to determine the direction of the user's gaze, and the distance of the point of gaze from the eye (whether near or far), so that the display system can optimize the generated holographic image for the precise focus of attention. The proposed AR display can render images at variable focal lengths, so it could be used for people with visual accommodation issues, thereby allowing them to participate in AR-supported education and training programs. The device could also have other possible uses in medical (such as better understanding of the human visual system) and training fields.

The two branches of this project, the holographic display, and the 3D eye tracker, are closely linked and each improved by the other. The 3D eye tracker utilizes an enriched set of signals and sensors (multiple cameras for each eye, and a multiplicity of infra-red (IR) LEDs), from which the system extracts the multiple tracking parameters in real time: the horizontal and vertical gaze angles, the distance accommodation, and the 3D position and size of the pupil's opening. The distance accommodation is extracted by analyzing Purkinje reflections of the IR LEDs from the multiple layers in the eye's cornea and lens. A neural network extracts the aforementioned 3D tracking results from the multiple sensors after being trained on a large body of ground truth data. The training data is generated from multiple human subjects who are exposed, instantaneously to known patterns on external displays at a range of distances and angles from the eye. Simultaneous to these instantaneous patterns, the subject is also shown images from the near-eye holographic image generator whose eye box location and size have been previously optically calibrated. One part of each pattern will be shown, instantaneously, on an external display and the other part, at the same instant, on the holographic display. The subject can only answer correctly a challenge question if they have observed both displays simultaneously. This can only occur if the eye is at a precise 3D location and also at a precise known gaze angle. The eye tracker will be further improved by integrated its training and calibration with the high precision (but very bulky) BinoScopic tracker at UC Berkeley, which tracks using precise maps of the user's retina. The holograhic image generator uses the real time data from the 3D eye tracker to generate holograms whose highest image quality is at the part of image that is currently on the viewer's fovea, and at the distance to which the user is currently accommodated. The image quality is improved by a trained neural network whose inputs are images from a camera placed, during training, at the position of the viewer's eye.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.