Edward H. Adelson - US grants

The measurement and use of visual motion is a fundamental component of biological and machine vision systems that provides essential sensory information for tasks such as navigation, object manipulation and recognition. Significant advances have been made toward understanding how vision systems might solve the individual problems of detecting sudden movements, segmenting the scence into distinct objects on the basic of motion discontinuities, tracking objects of interest, recovering the three-dimensional structure and movement of object surfaces, and inferring their own movement relative to the environment. This research examines how solutions to these problems are integrated into a motion analysis that performs these functions with speed, accuracy, reliability and flexibility. Such a system must embody multiple computational strategies that combine fast and robust methods for deriving qualitative motion information with slower, accurate methods for deriving quantitative models of three- dimensional structure and motion. The approach taken in this project brings together theoretical analyses, implementation and testing of computer algorithms, and observations on human motion perception. This research will lead both to significant improvements in the performance of computer vision systems at analyzing dynamic images, and new understanding of motion analysis in the human visual system.

DESCRIPTION (adapted from the proposal abstract): The proposed research is to conduct experimental and theoretical research on the problem of integrating form perception and the perception of motion. The proposed work uses image segmentation to get better model predictions of perceived motion and uses motion perception as an indication of image segmentation. The experiments will employ simple computer-generated stimuli that reveal a variety of interesting perceptual effects. Among them will be rotating ellipses that appear non-rigid, translating rhombus figures that appear to move incorrectly, and reduced variants of barber pole figures. Each stimulus has been chosen to study certain fundamental issues in motion analysis, and to test various properties of the candidate motion models. Modeling will proceed in parallel with the experiments. Existing models and new ones will be tested and modified in accord with the experimental results.

DESCRIPTION (Adapted From The Applicant's Abstract): The proposed research will contribute to the understanding of human lightness perception. Recent work on lightness perception has suggested that many phenomena may be understood in terms of anchoring and perceptual grouping. The essential ideas are: (1) The luminance distribution of a scene serves to anchor and scale local luminance measurements into a particular lightness mapping. (2) Perceptual grouping of luminance patches establishes boundaries that affect the spatial weighting scheme of the lightness mapping. The investigators will perform experiments on lightness perception to test the existing models, to help formalize the notions of anchoring and grouping, and to determine how models may be modified or improved in order to account for a wider range of phenomena. The anchoring problem is conceptually recast in a framework of "atmospheres." This approach seeks to estimate lighting and viewing condition parameters from luminance statistics and configurations in order to establish mappings between image luminances and perceived reflectances. The proposed experiments will employ computer-generated and physical stimuli to measure lightness mappings under a wide variety of conditions. Stimuli will be tested both in traditional displays and in specially-designed panoramic displays that permit control of all image luminance values. Anchoring experiments will investigate statistical aspects of luminance distributions that are critical in determining the lightness mapping. Grouping experiments will test the importance of particular junction and contour configurations in establishing boundaries for lightness perception. Additional experiments will investigate the effects of naturalistic cues in everyday scenes in order to identify further mechanisms that contribute to lightness perception. The results of these investigations should be of interest to researchers in lightness, color, and perceptual organization.

Even the best computer vision system falls far short of human vision. A person has no problem recognizing metal as metal or wood as wood. These are examples of material perception in which vision relies on combinations of color, texture, transparency, and glossiness to recognize the material surface of objects. With NSF Support Dr. Edward Adelson studies how perception puts information together, thereby allowing material perception. His working hypothesis is that human vision relies on certain statistical relationships between colors and patterns within an image, and uses these to infer the material. However, the exact nature of these relationships remains to be understood.
Broader impacts of the research are important in everyday life. People care greatly about the appearance of their skin and hair, the clothing they wear, and the food they eat. If we can understand the principles that determine material perception, it will help industrial researchers who seek to make new products with improved surface appearance, such as new kinds of cosmetics or paint. An understanding of material perception may also pay off in better machine vision systems. For instance, an automated vehicle should be able to distinguish pavement, dirt, mud, or ice and adjust its driving accordingly, but today's machine vision systems find such problems quite difficult. By emulating the mechanisms of human vision, we can develop more powerful machine vision systems.

DESCRIPTION (provided by applicant): The goal of this program is to support graduate training in the field of vision at MIT. Funding for four predoctoral students in the first year is requested, with an increase of one per year for the next four years. The Department of Brain and Cognitive Sciences (BCS) has a very broad range of research and teaching in the vision sciences, with particular expertise in visual development, neurophysiology, psychophysics, and computational models. In conjunction with participants from the Department of Biology and the Department of Electrical Engineering and Computer Science, the program will provide excellent opportunities for students to explore a wide range of topics in vision. In the first year of the graduate program, all BCS students take two of three core courses: Systems Neuroscience in the fall, and either Cognitive Science or Cellular and Molecular Neurobiology in the spring. During the first year, students also begin research, either immediately or after one to three rotations in laboratories. Throughout the course of their education, graduate students are expected to conduct research at least half-time, while taking courses or teaching, or full-time during summers and after coursework is complete. Acting as teaching assistants is required for three terms, usually one term each in the second, third, and fourth years. An advisory committee is chosen within the first year, although the primary advisor usually has the greatest contact with the student. Written and oral qualifying exams take place after the second year, and a research report is required during the third year. The BCS graduate program is coordinated by a Graduate Committee comprised of representatives from each area of the department, and a Graduate Office which implements the Committee's decisions and administers the graduate program. The purpose of this training program is to produce a new generation of vision researchers who are highly trained in their own field of expertise, but who are also broadly educated about the scientific foundations that link the vision community together. Relevance: Vision scientists seek to understand how the eye and brain work, and also seek to understand what goes wrong in visual disorders and diseases. Today's vision scientists must be trained in multiple disciplines, including biology, psychology, neuroscience, and computer science. MIT's training program is designed to give students the scientific breadth and depth that they need to make advances in the field.

In recent years it has become easy for anyone to digitally manipulate images using software such as Adobe Photoshop. This puts powerful creative tools in the hands of photographers. Unfortunately, it also makes it easy to tamper with images with ill intent. The digital forger can provide false information to the news media and can alter evidence in a legal proceeding.

In this project, we will collaborate with researchers at Adobe to develop methods for detecting forgeries. When we examine the pixel patterns within genuine images, we find certain statistical regularities due to the way the images are formed. The forger disturbs these regularities, giving us the opportunity to detect the subtle statistical traces left behind. We will investigate image statistics related to lighting, color, contrast, and perspective.

DESCRIPTION (provided by applicant): Vision provides us with information about the objects in the world around us;it also allows us to see the materials that they are made of. Material perception is important: for example, it lets people see whether a sidewalk is icy and it permits a physician to decide whether a mole looks dangerous. At present, very little is known about material perception. This project will study at both a theoretical and empirical level. The research will assess the importance of basic factors such as visual resolution in making material judgments. The material judgments can range from simple descriptions of appearance (e.g., "how shiny is this surface?") to more complex judgments about the material properties (e.g., "how soft is this carpet?") The answers will provide constraints for models of material perception, and will also help in understanding the impact that various visual deficits will have on a variety of tasks. There are losses in visual information that occur when materials are viewed on a computer monitor rather than seen in the real world. There are other losses that occur due to eye disease. By measuring the impact of specific kinds of information, the project will indicate the best directions to go in modeling the underlying perceptual mechanisms. Those mechanisms will also be studied in other ways. By recording eye movements, the researchers will learn the local image features that subjects fixate on when making material judgments. In other experiments, specific features based on the outputs of wavelet-like filters will be evaluated as potentially useful sources of information. If a candidate feature (e.g., the skewness of a filter output) is important to humans, then by manipulating this feature it should be possible to alter a material's appearance in predictable ways. PUBLIC HEALTH RELEVANCE This project will help determine the visual mechanisms underlying material perception, which includes the perception of visual qualities like glossiness and more physical qualities like wetness or slipperiness. Material perception is of widespread importance, and when vision is impaired (by eye disease, or by limitations in digital displays) material perception is degraded. Understanding the basic mechanisms will help improve the visual performance of digital systems such as those used in telemedicine;it will also help understand the impact that eye diseases have on simple tasks such as avoiding icy patches on the sidewalk.

DESCRIPTION (provided by applicant): The visual perception of materials is a basic part of human vision, but is currently not well understood. People depend on material perception constantly in daily life. Examples are in navigation (Am I about to step on an icy patch?), eating (Is this cream cheese moldy?), mate selection (Does my date have healthy looking skin?), and medical diagnosis (Is this a suspicious looking mole?). Prior research in material perception has mainly used simple, controllable stimuli. This project will study a diverse range of naturalistic stimuli, such as occur in the real world. In order to do so, novel techniques that are quite different from those commonly used in material perception will be developed. New image databases will also be developed. In one set of experiments, subjects will see objects of a fixed shape made of different materials, and the researchers will assess the subjects'ability to judge and describe material qualities. In another set of experiments, subjects will view images from a finite set of categories, (for example, plastic, paper, cloth, or metal) and the researchers will measure the speed and accuracy of categorization. There is evidence that human observers can extract material information extremely rapidly, the experiments will quantify the course of this capability. In other experiments, subjects will see images that are degraded in various ways, which will elucidate the importance of various factors such as color, contrast, and detail in material perception. The project will provide a foundation for future work on material perception in the real world. An understanding of these basic issues would help in understanding some practical visual problems. For example, as people get older, their vision degrades, making it more difficult to detect slippery patches of floors or sidewalks. A science of material perception would help in developing recommendations on the lighting, layout, and materials used in sidewalks and corridors. In another example, when a physician is evaluating a patient in telemedicine (i.e., viewing the patient over a video link), different kinds of image degradation will lead to different difficulties in judging the appearance of, say, a wound, a rash, or a mole. A science of material perception could lead to improvements in video quality that are essential to the telemedicine setting. PUBLIC HEALTH RELEVANCE: The visual perception of materials is an essential visual capability. Degraded vision limits the ability to make basic judgments about the slipperiness of a sidewalk, the freshness of food, or the health of skin. Little is known about these capabilities, and the proposed research will establish some of the foundations needed for a theoretical and practical understanding of material perception.

This project seeks to develop tactile sensing technology that emulates many qualities of human skin. The goal is a sensor that can determine the texture and shape of objects that it touches, as well as the forces distributed across the surface. The new sensor is made of a block of clear elastomer, with a compliance similar to that of the human fingertip, covered with a flexible reflective skin. A small light source and a camera are embedded in the device. When an object contacts the skin, the surface is distorted, leading to a change in the reflected light pattern. Machine vision techniques convert the patterns into estimates of the forces on the skin. The project is testing a number of optical and mechanical designs, and is developing the corresponding image analysis techniques, in order to characterize and optimize the performance. Because the sensor is compliant, it can be built into a human-like robotic finger, providing gripping surfaces that are mechanically stable as well as highly sensitive. The new technology may also be useful in medical applications such as minimally invasive surgery, where it is important for the surgeon to sense the mechanical properties of the tissues that are being explored.

People care greatly about the appearance of translucent materials such as food, skin, soap, and marble, and they are able to distinguish subtle differences in these materials based on their appearance. The translucent appearance of these materials is caused by internal volumetric scattering, which is challenging to simulate, especially because humans are so sensitive to their subtleties. Since in the natural world scattering materials are the norm, not the exception, it makes sense that the human visual system is so well engineered to analyze them. However, very little is known about how this analysis is achieved because the perception of volumetric translucency is almost unstudied.

This collaborative project, involving faculty from three universities with complementary expertise in computer graphics, human vision, machine learning, and computer vision, addresses the fundamental unsolved problem of understanding translucency for graphics. The PIs will develop a perceptually-motivated pipeline for translucency, contributing new scattering representations, perceptual dimensions, and computational algorithms to computer graphics. The scattering representations, based on a polydispersion model, will provide analytic expressions for wavelength-dependent bulk scattering properties of translucent media; this will significantly expand the range of materials that can be simulated with high visual fidelity. Finding perceptual knobs that relate physical scattering parameters with visual appearance will be achieved by coupling large-scale computation (using cloud computing) with controlled perceptual studies. Novel acquisition approaches that employ hyperspectral imaging will be created, as will editing and rendering applications that use the new perceptual representations of translucency. Low-dimensional models to represent scattering media will be developed and used to enable efficient and accurate acquisition and rendering. A suite of test materials and scenes will be developed to evaluate the fidelity of rendered images based on the developed theory and computational applications.

Broader Impacts: Currently, the simulation of translucency presents challenges in terms of both computation and visual fidelity. This restricts the ability of practical algorithms to predictively simulate translucent materials, thus fundamentally limiting the use of graphics in real applications. By building the computational tools to characterize, study, and use knowledge of translucency perception, this research will fundamentally change the graphics pipeline for translucent materials. and will potentially revolutionizing industrial design, interior design, skin care and cosmetics, and entertainment.

The project includes an education program that is tightly coupled to the research program. The PIs have already been meeting twice a week for more than six months, and their graduate students already share data, code, and equipment. During the activity, the students will make week-long and month-long visits to each other's laboratories to collaborate, and in this way the project will produce a generation of researchers who are "T-shaped" in the sense of being both deep in their respective fields and able to work effectively across these synergistic disciplines. The PIs also plan to organize a workshop that will brins together researchers in vision science, computer graphics, and computer vision, so that the important ties between these fields are strengthened even further.