Alan Gilchrist - US grants

Recent progress in the study of how we perceive black, gray, and white surfaces has been based heavily on the insight that the retinal image is encoded in terms of relative, rather than absolute, light intensities, or luminances. This research will investigate two serious problems for models based on relative luminance, known as ratio models. In one study, reports by observers in a completely homogeneous visual field that grows either brighter or darker for an extended period of time, at a rate too slow for detecting the change, will indicate whether the visual system senses absolute luminance in the absence of visible spatial and temporal change. The homogeneous visual field will eliminate spatial changes in luminance and the gradual change of luminance over time will eliminate the detection of temporal change. The hypothesis that only relative luminances are encoded suggests the counter-intuitive result that under these conditions the human observers will not be able to discriminate a very bright field from a very dark field. Deduction of visual processes underlying surface color perception requires a specification of whether relative or absolute luminances, or both, are available to the visual system. In the second study, observers will view various displays through a glass panel that reflects a homogeneous sheet of light, known as a veiling luminance; their experience will be much like looking through a fog. Although contrasts, or relative luminances, are dramatically reduced under these conditions, observers are able to identify gray shades correctly if the display behind the veil is sufficiently complex. Systematic variation of the display will enable the determination of factors that allow correct identification. These results should carry strong implications regarding how relative luminances are processed so as to produce visual experience of white and black surfaces.

This research is part of a broad effort to determine the means by which the human visual system processes the pattern of light falling on the retina, in order to determine the white, gray or black level of surfaces within a person's field of view. It is generally accepted that perceived surface gray levels are based on relative light intensities extracted from intensity gradients in the retinal image, but little is known about how these relative intensities are anchored, that is, processed to yield the specific shades of gray that we see. Strictly speaking, relative intensities provide information only about relative shades of gray. Absolute shades of gray can be derived only with the addition of an anchoring rule. For instance, in many scenes the visual system automatically treats the highest intensity as white. This can be demonstrated under the simplest conditions by placing an observer's head inside a large dome, half of the interior of which is painted black and half of which is painted gray. The gray part will appear white and the black part will appear gray, consistent with the highest-intensity-as-white rule. However, if an observer's head is placed within a uniformly painted black dome containing a white disk at its center, the black dome appears white and the disk appears luminous. This would follow from a rule by which white is assigned to the surrounding region, not to the highest intensity. In this research, human observers will be placed in a variety of visual environments, chosen to distinguish different potential anchoring rules. These will include simple patterns on large plastic domes that completely surround the viewer, more complex lab displays, computer simulated displays, and on-site natural scenes. Analysis of how the anchoring rules change as the visual field becomes increasingly complex may well fill an important gap in the modelling of surface color perception.

9514679 GILCHRIST This research will further examine how the human visual system assigns black, white and gray values to surfaces represented in the image projected onto the retina. This problem remains unsolved, in human vision as in computer vision, because the intensity of light reflected from any given surface in the visual world, regardless of its actual shade of gray, can vary by a factor of a billion to one, depending on lighting conditions. Prior research has established that the perceived shade of gray of a surface depends on the intensity of that surface relative to that of neighboring surfaces, not its absolute intensity, and that relative intensity is just what is encoded when light from the world strikes the human retina. But this brings into focus what has been called the anchoring problem: How is an absolute or specific shade of gray assigned to these relative intensity values? The general approach of the work will be to formulate alternative potential rules the visual system might use and then to construct visual displays that produce retinal images capable of testing among these rules. For a given display, one rule might predict that a particular surface will be seen as middle gray while a competing rule might predict that the same surface will be seen as white. The display will be presented to a set of naive human observers, from whom reports will be obtained as to the apparent gray shade of the surface. They will indicate the appearance of the target surface by choosing a matching chip from a standard scale of gray chips under standard lighting conditions. The work of this project will build upon a series of findings from previous funding that have established the rules of anchoring for relatively simple visual displays. In short these rules state that, for simple images, the surface with the highest intensity will be seen as white and darker regions will be scaled relative to this standard. But relative size of regions has also been shown to be important and can override the role of the highest intensity in certain cases. Prior work along these lines has produced a model of visual functioning that has already proven to account for much of the research literature in this field. The work of this research project will be guided by contradictions and gaps in this model. An important goal will be the extension of the model to ever more complex images. The solution of this problem in human vision should make it easier to solve the problem for computer vision. ***

The goal of the proposed project is to determine how the human visual system computes the black, gray and white values of object surfaces represented in the optical image projected onto the retina. This problem has not yet been solved for machine vision. For example, there is currently no program that can determine the surface colors of objects within a video image. A key part of the problem lies in disentangling variations in image intensity that are due to light and shadow from those that are due to light and dark gray surfaces. The work proceeds using the method of psychophysics. Visual images or displays are created that, when viewed under controlled conditions, allow a test between competing theories or hypotheses. Ideally, for example, an image might be created so that a given target surface within the image should appear white according to Theory A but black according to Theory B. The image is then viewed by human observers, who report the appearance of the target surface.
Three years ago, the PI proposed a new theory based on the concept of anchoring. According to this theory, a complex image is parsed by the visual system into frameworks, or perceptual groups, based on rules of grouping. For any given surface, several lightness (gray scale) values are computed, one for each group to which it belongs. The shade of gray actually perceived for that surface is a weighted value of each of these computed values. The work proposed here will extend the theory into a class of illusions that involve spatial intensity gradients. The work will also focus on certain experience effects: how the perceived gray value is influenced by prior exposure to related scenes. Finally, a series of experiments will examine how the relative area of surfaces influences their perceived gray level, especially in more complex images. The project will also include the creation of a web page containing a comprehensive collection of visual illusions in lightness. This publicly accessible collection is likely to be well used by educators and other researchers.

The project revolves around the idea that learning physics is the most effective in an interactive learning environment involving two related components that are usually kept separate: a real-world experiment and a computer-based visualization. In this project, we will create a learning environment that combines microcomputer-based laboratories (MBLs) with immersive virtual-reality technology. While MBLs will provide students with the possibility of real-world experimenting and data collection, supported by real-time graphing tools, virtual reality technology will be used to support the visualization of both visible and invisible physical processes and abstract concepts (e.g., vectors of forces, field lines or energy levels) underlying the same experiment students observe in MBLs. The research objective of the current project is to investigate how different aspects of computer-based visualization, supported by immersive virtual reality and MBLs, affect and interact with student-generated visual/spatial representations and students' qualitative understanding of abstract physics phenomena.

In terms of broader impacts, the project will bring new insights as to how to use visualization to facilitate student' conceptual understanding. In addition, the project will give a theoretical basis for design and evaluation of educational materials involving different visual/spatial representations. Tools and materials we develop in the project will make it easier for faculty around the country to develop students' visualization skills and qualitative reasoning in introductory physics classes.

The educational objective of the proposal is to develop educational software and curriculum materials in the area of introductory mechanics with a focus on visualization processes, which allow students to participate actively in their own learning and to construct scientific models.

How we distinguish black, white and gray surfaces has yet to be explained scientifically. With NSF support, Dr. Alan Gilchrist studies how the visual system determines the shades of gray in the world we see. Consider the follow: Under the same lighting, a white paper reflects more light to the eye than a black paper. But a black paper under brighter illumination can reflect much more light to the eye than a less- illuminated white paper. Nevertheless we see black as black and white as white. This is the kind of puzzle that Dr. Gilchrist attempts to understand. His studies progress from the simplest possible scenes. An observer's head is placed inside a large, opaque hemisphere that fills the observer's visual field. The inside surface divides vertically into two gray shades. Regardless of which shades are used, the lighter one always appears white. This is an important clue to the visual system. In these simple circumstances, it appears to assign white to the brightest part of the scene, but the relative area of the two regions, and the difference between their actual gray shades also play a role. Next to be studied are laboratory scenes that contain more than one region of illumination. Here the rules become more complex. The final studies in the series include complex, natural scenes using a new apparatus that can insert an artificial patch of gray onto any surface of the scene being viewed. This innovative use of technology sets the funded work apart from previous efforts. It allows investigators of lightness perception to conduct research on the richly complex surfaces that constitute real-world scenes. One broad impact of the work will be to make this technology and an illusion archive available to other scientists. In addition, understanding how gray shades are computed correctly in human vision will lead to programming advances in machine vision.

Though it might surprise the average person, human vision has not yet been explained scientifically. For example, we do not yet have enough information about vision to design a robot that can actually see. No robot or computer exists that can determine the size or even the color of an object in its field of view. Although the perception of size and color is easy for human vision, we do not know what kind of "software" the brain uses to do this. The open questions addressed in this project include how the human visual system detects lightness: the black, white, or gray shade of objects in the visual field, and especially how this detection depends on the perceived distance of that surface. A second issue in this research is methodological: Researchers testing lightness perception using paper of varying lightness have found different results from those found by researchers using computer monitors. Obviously, a theory of visual phenomena would not expect such differences.

With support of the National Science Foundation, Dr. Alan Gilchrist will examine the nature of the software of lightness perception. In a series of experiments, Dr. Gilchrist will explore the relations between surface lightness, lightness context, illumination, and surface distance. All the experiments will be conducted in two ways: (1) using actual 3D displays made from paper, and (2) using simulated displays created with CRT images in a mirror stereoscope. This work will clarify a series of nagging and controversial questions concerning the nature of the relationship between lightness perception and depth perception (two of the most important visual dimensions) and between lightness perception and perception of the illumination level. The resulting understanding will have a number of impacts. It will help guide neuroscientists in what to look for in the brain, both in the study of vision itself, and in the study of cognitive processes more generally. The work also will benefit machine vision, by laying the groundwork for successful machine-vision algorithms, thereby enhancing our technical capabilities. The project will also provide a needed service to the field of vision by advancing the integration of two bodies of research on lightness, those using traditional paper and light displays, and those using stereo CRT images.

Dr. Alan Gilchrist at Rutgers University will conduct a series of experiments to discover how the human visual system can accurately determine the gray shade of visible surfaces even when the contrast in the image reaching the eye has been severely reduced. This occurs when a scene is viewed through a sheet of glass. Light reflected off the front surface of the glass (e.g., a reflection of the sky on a car windshield) combines with the pattern of light coming through the glass, reducing the contrast of that pattern. A bright glare source in the visual field can have the same effect, due to intense scattered light within the eyeball. The experiments will systematically isolate image features that are present in 3D scenes (where it is known that humans can successfully correct for such reflected light) and 2D patterns (where the correction fails). These features include shadows, light-absent crevices, and certain patterns where overlapping edges intersect. In addition, the perception of colored surfaces overlain with colored reflections will also be studied.

The human visual system is able to determine the gray shade of objects despite changes in illumination level, changes in the background behind the object, and changes in the media that intervene between the viewer and the object. This latter problem has been almost totally neglected by vision research. A solution will advance our understanding of how the brain computes the gray shade of visible objects. It is widely agreed that in order to determine the gray level intensity of a surface, the human visual system relies crucially on the strength of contrast at the edges of that surface and neighboring surfaces. When the entire scene containing that surface is overlain with reflected light (or light scattered from a glare source), the strength of contrast at those edges is dramatically reduced. Nevertheless, when the scene is 3D and somewhat complex, the human eye can automatically disentangle the reflected light from the scene itself, and perceive the gray shade of the surface correctly. No machine visual system can do that. Knowledge about how the human visual system achieves this feat will contribute to the enhanced ability to program a machine to replicate it.

The two leading theories of how the white/gray/black shade of a surface is computed in your brain will be tested against each other in a series of 10 experiments. Human observers will be presented with three-dimensional displays carefully constructed so that, under controlled viewing conditions, the two theories make different predictions regarding the gray shade that will be perceived for a test surface embedded within the display. All of the displays will incorporate a serious challenge for vision, namely, multiple areas of light and shadow. These displays will vary in terms of (a) whether the change of illumination is projected onto a surface or occurs at a corner (or bend) in the surface, (b) the number of different gray shades on each surface, and (c) the range of gray shades on a given surface. The observers will be asked to match critical parts of each display for gray level and level of illumination. The strengths and weaknesses of each of the theories will be evaluated based on the observer matches. Ideally these results will suggest how the respective strengths of the two theories might be integrated.

How humans see the white, gray, and black shade of objects has not yet been explained scientifically. The basic problem is that the light that an object reflects to the eye does not reveal the shade of the object because it depends so heavily on the intensity of illumination on the object. Thus any intensity of light can be reflected from any shade of gray. The problem can only be solved by analyzing the surrounding context. Advances in our understanding of the nature of this analysis are absolutely necessary for guiding brain research in this area. The work will impact computer vision as well. At the moment, no robot can simply look at an object and report its shade of gray. Work with human vision provides the best hope for making better computer programs. Ironically, the proposed work will be conducted, not using computer images, as is almost universally done in human vision research, but with actual three-dimensional displays that are far more realistic.