Ruth Rosenholtz - US grants

Visual clutter is something many of us encounter everyday, whether it is the clutter of items on a desktop, real or virtual, the clutter of signs on a busy street, or the clutter of objects on x-ray screening display. Our intuitions notwithstanding, we know surprisingly little about how to define visual clutter scientifically. We can judge it subjectively, but the problem of formulating an explicit definition has gone unsolved.

With NSF funding, Dr. Rosenholtz is using what we know about visual perceptual systems to develop a mathematical model that can estimate the degree of clutter in a visual display. The model is based on the simple notion that clutter begins when it becomes difficult to add items to a display without degrading performance. There is a counterintuitive consequence to this idea, which is that clutter is not a simple function of the total number of items. Instead, clutter is due to the complex interplay of a number of items with other factors such as item arrangement, shape, and color. The model will be tested against measures of clutter as perceived by human observers, in the context of a variety of visual displays. The measures include subject judgments as well as a number of established human performance metrics. The results of these experiments will advance our understanding of how visual scenes are perceived, and these advances may help with efforts to reduce clutter in displays where such efforts are helpful if not critical.

DESCRIPTION (provided by applicant): Identifying a visual stimulus can be substantially impaired by the mere presence of additional stimuli in the immediate vicinity. This phenomenon is called "crowding," and it powerfully limits visual perception in many circumstances, especially in the peripheral visual field. There is a rich body of literature detailing the phenomenology of crowding, but we do not know why crowding occurs. We lack a computational model that can predict what information will be available to an observer in an arbitrary crowded display. A popular hypothesis is that crowding results from obligatory "texture processing," but there have been few efforts to formalize and test what this might mean, despite broad agreement that crowding reflects some form of "excessive integration." Dr. Rosenholtz has extensive experience with computational models of texture processing, which are a powerful means of defining the exact nature of "texture processing" and testing the ability of such models to explain and predict visual behavior. The proposed research has 3 aims: (1) To clarify and formalize the hypothesis that crowding is due to a "texture" - i.e. statistical -- representation of the crowded stimuli. (2) To collect behavioral data from a wider variety of displays and tasks than is typically studied in crowding. (3) To develop and validate the first general-purpose model of visual crowding. To achieve these aims, Dr. Rosenholtz will apply state-of-the-art computational tools for texture synthesis to "crowded" stimuli. "Texturizing" crowded arrays of stimuli affords a tool for visualizing the information available in a crowded display and a vocabulary for describing its representational content. Thus, Dr. Rosenholtz will attack the problem of crowding through a useful synthesis of computer graphics, computer vision, and psychophysics. PUBLIC HEALTH RELEVANCE: Understanding crowding, besides elucidating representations and performance of normal human vision, is crucial for disorders like age-related macular degeneration, for which, without foveal vision, virtually all perception is essentially crowded. In addition, percepts under crowding may be related to percepts under other visual dysfunctions where there is "excessive integration", such as amblyopia and simultagnosia. Successfully predicting crowding severity would also advance the design of low-vision aids for older adults and improve our ability to design for the visually-impaired.

DESCRIPTION (provided by applicant): Vision provides us with information about the objects in the world around us;it also allows us to see the materials that they are made of. Material perception is important: for example, it lets people see whether a sidewalk is icy and it permits a physician to decide whether a mole looks dangerous. At present, very little is known about material perception. This project will study at both a theoretical and empirical level. The research will assess the importance of basic factors such as visual resolution in making material judgments. The material judgments can range from simple descriptions of appearance (e.g., "how shiny is this surface?") to more complex judgments about the material properties (e.g., "how soft is this carpet?") The answers will provide constraints for models of material perception, and will also help in understanding the impact that various visual deficits will have on a variety of tasks. There are losses in visual information that occur when materials are viewed on a computer monitor rather than seen in the real world. There are other losses that occur due to eye disease. By measuring the impact of specific kinds of information, the project will indicate the best directions to go in modeling the underlying perceptual mechanisms. Those mechanisms will also be studied in other ways. By recording eye movements, the researchers will learn the local image features that subjects fixate on when making material judgments. In other experiments, specific features based on the outputs of wavelet-like filters will be evaluated as potentially useful sources of information. If a candidate feature (e.g., the skewness of a filter output) is important to humans, then by manipulating this feature it should be possible to alter a material's appearance in predictable ways. PUBLIC HEALTH RELEVANCE This project will help determine the visual mechanisms underlying material perception, which includes the perception of visual qualities like glossiness and more physical qualities like wetness or slipperiness. Material perception is of widespread importance, and when vision is impaired (by eye disease, or by limitations in digital displays) material perception is degraded. Understanding the basic mechanisms will help improve the visual performance of digital systems such as those used in telemedicine;it will also help understand the impact that eye diseases have on simple tasks such as avoiding icy patches on the sidewalk.

DESCRIPTION (provided by applicant): The visual perception of materials is a basic part of human vision, but is currently not well understood. People depend on material perception constantly in daily life. Examples are in navigation (Am I about to step on an icy patch?), eating (Is this cream cheese moldy?), mate selection (Does my date have healthy looking skin?), and medical diagnosis (Is this a suspicious looking mole?). Prior research in material perception has mainly used simple, controllable stimuli. This project will study a diverse range of naturalistic stimuli, such as occur in the real world. In order to do so, novel techniques that are quite different from those commonly used in material perception will be developed. New image databases will also be developed. In one set of experiments, subjects will see objects of a fixed shape made of different materials, and the researchers will assess the subjects'ability to judge and describe material qualities. In another set of experiments, subjects will view images from a finite set of categories, (for example, plastic, paper, cloth, or metal) and the researchers will measure the speed and accuracy of categorization. There is evidence that human observers can extract material information extremely rapidly, the experiments will quantify the course of this capability. In other experiments, subjects will see images that are degraded in various ways, which will elucidate the importance of various factors such as color, contrast, and detail in material perception. The project will provide a foundation for future work on material perception in the real world. An understanding of these basic issues would help in understanding some practical visual problems. For example, as people get older, their vision degrades, making it more difficult to detect slippery patches of floors or sidewalks. A science of material perception would help in developing recommendations on the lighting, layout, and materials used in sidewalks and corridors. In another example, when a physician is evaluating a patient in telemedicine (i.e., viewing the patient over a video link), different kinds of image degradation will lead to different difficulties in judging the appearance of, say, a wound, a rash, or a mole. A science of material perception could lead to improvements in video quality that are essential to the telemedicine setting. PUBLIC HEALTH RELEVANCE: The visual perception of materials is an essential visual capability. Degraded vision limits the ability to make basic judgments about the slipperiness of a sidewalk, the freshness of food, or the health of skin. Little is known about these capabilities, and the proposed research will establish some of the foundations needed for a theoretical and practical understanding of material perception.

DESCRIPTION (provided by applicant): In our daily lives, we search for our keys, look for a friend in a crowd, and try to find a button on our remote control. Visual search is an important part of many applications, as well, including search for a possible tumor in a mammogram, or search for a threat in a baggage x-ray. Search has been studied intensely for 30 years, but the results remain puzzling. Some searches are easy, even when the target appears against a background of many distractor items. Other searches become quite difficult with many distractors, even when the target and distractors are quite distinct. We lack a computational model that can predict which searches will be easy or hard, or make quantitative predictions of search performance for arbitrary displays. The overall goal of the proposed research is to better understand visual search based on the insight that the capabilities of peripheral vision provide a fundamental constraint on search performance. Peripheral vision enables fast target detection in the periphery (the target seems to pop out), and guides eye movements until, ultimately, the observer finds the target. The proposed work builds on recent modeling of peripheral vision. These recent results suggest that peripheral vision processes not individual items, but rather sizable local patches, which it represents in terms of a rich set of summary statistics (Balas, Nakano, & Rosenholtz, 2009). The proposed research has two intertwined aims. Aim 1 is to develop and test models of visual search based on the hypothesis that search is constrained by the discriminability of peripheral patches containing a target (and a number of distractors), and those containing only distractors. In particular, Dr. Rosenholtz will examine the extent to which search performance can be predicted by: Peripheral discriminability of (1) individual items (2) larger, crowded patches. (3) Predicted discriminability of target present vs. target absent patches based upon their summary statistic representation; (4) A quantitative model of the fixations required to find a target. Aim 2 is to test whether a wide range of search phenomena can be accounted for by a single mechanism of peripheral vision. In particular, Dr. Rosenholtz will examine: (1) The predominance of search asymmetries, e.g. that it is easier to search for a 'Q' among 'O's than for an 'O' among 'Q's; (2) Differences between search for a target differing from distractors by a single feature, by a conjunction of features, and by a configuration of basic features; (3) Somewhat puzzling accounts of what constitutes a basic feature that can guide search; (4) The effects of grouping on visual search. PUBLIC HEALTH RELEVANCE: Visual search is a near-ubiquitous task in our daily lives, and understanding it will clarify more generally the processes by which we constantly move our eyes to piece together information about the world. In addition, understanding visual search will elucidate representations and performance of normal human vision. Successfully modeling visual search will shed light on important search tasks such as finding a tumor in a mammogram, and will enable improved design of low-vision aids for older adults and the visually-impaired.

Peripheral vision comprises over 99.99% of the visual field. Its strengths and limitations strongly constrain visual perception -- what humans can see at a glance, and the processes by which they move their eyes to piece together information about the world. Peripheral vision differs from foveal vision in complex and interesting ways, most importantly due to "crowding," in which identifying a peripheral stimulus can be substantially impaired by the presence of other, nearby stimuli. This project will examine the nature of the encoding in visual cortex, through development and testing of a set of models of peripheral vision. These models will be targeted at answering key questions about the neurobiological mechanisms. The collaborating investigators, in the US and Germany, will develop models and create a benchmark dataset of behavioral results to be explained. The models and dataset will be made freely available, to aid other researchers and to inform the development of applications such as heads up displays and user interfaces. This work will provide insight into what features are encoded in visual cortex, as well as what tradeoffs may have led the visual system to develop that encoding. Understanding those tradeoffs may inform computer vision which, like human vision, faces constraints on processing capacity.

The development of new model variants will be based on insights from neurophysiology, natural image statistics, sparse coding, and the recent success of convolutional neural networks in artificial intelligence. The investigators will gather benchmark behavioral phenomena far richer than existing crowding datasets, through a combination of studying natural image tasks and model-driven experiments. They will then compare predictions of the new models, as well as of Dr. Rosenholtz's existing high-performing model of peripheral vision, on the benchmark dataset. Doing so will identify the best-performing model(s), and answer key questions about the nature of pooling computations and of non-linear operators, and about the complexity, nature, and purpose of the features encoded by peripheral vision.

A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF).

Most of vision is peripheral vision. The central fovea comprises only 1.7 degrees of the visual field, leaving 99.99% of the visual input to fall on the peripheral retina. Peripheral vision differs from central vision in complex and interesting ways, most importantly due to 'crowding', in which identifying a peripheral stimulus can be substantially impaired by the presence of other, nearby stimuli. Crowding is a critical bottleneck in vision; substantial behavioral evidence demonstrates that crowding greatly limits our ability to perform most real-world visual tasks. The research team will pit a dominant class of models, known as pooling models, against a rich corpus of recent experimental work which has seemed to suggest that visual mechanisms might be more complicated and dynamic than previously thought. This work could enable development of improved materials and tools for those with age-related macular degeneration, in which patients lose central vision. It could also aid design of heads-up displays, user interfaces, and information visualizations such as situation maps for military decision-making.

Pooling models posit that crowding arises from averaging of features over large local regions, fixed in size, which grow linearly with distance from the point of fixation. The PI has spent the last decade developing the computational and behavioral methodologies that allow one to derive quantitative, testable predictions from a state-of-the-art pooling model. Using these tools, the PI will examine a definitive set of crowding phenomena that have appeared to challenge a unifying account in terms of a pooling mechanism. She will take a three-pronged approach: 1) using modeling and behavioral experiments to examine the assumptions underlying the model challenges; 2) using behavioral experiments to eliminate possible confounds; and 3) designing new experiments to directly test alternative theories. The expected overall impact is to fundamentally advance mechanistic understanding of a key bottleneck in peripheral processing, and thus in visual processing more generally.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.