Wilson S. Geisler - US grants

The research proposed here is directed toward developing an understanding of the mechanisms of human visual sensitivity. Specifically, the proposed research will investigate, within both the rod and cone systems, the mechanisms underlying, (1) intensity discrimination in the dark-adapted eye, (2) the effects of background adaptation on intensity discrimination and apparent brightness, and (3) the effects of adaptation on spatial vision. On the basis of our previous work in areas (1) and (2), we have developed a quantitative theory of intensity discrimination and adaptation that is closely aligned with current knowledge of the electrophysiology and retinal neurons. This theory consists of a cascade of linear and static nonlinear stages. A number of psychophysical experiments will be carried out to extend our knowledge about areas (1) - (3) above and to test and extend our current theory. These include measurement of (a) increment- threshold functions at very high flashed-background levels in the dark- adapted rod and cone systems, (b) rod increment-threshold functions under various fixed levels of light adaptation, (c) brightness-matching functions at fixed levels of light adaptation or in the rod and cone systems, (d) increment-threshold functions during very early dark and light adaptation, (e) the sensitization effect during early dark adaptation and (f) contrast sensitivity functions (CSF's) under fixed states of light adaptation.

The goal is to udnerstand the basic mechanisms of visual sensitivity underlying spatial, contrast, and color discriminations. ONe important strategy in the research program is and has been to rigorously work out the implications for visual sensitivity of various anatomical and physiological mechanisms. These analyses lead to hypotheses and theories that can be tested with psychophysical experiments. Our aim is to work out what aspects of spatial, contrast and color discrimination can be accounted for by the properties of light (quantal fluctuations), the optics of the eye, and the sizes, lattice positions, and spectral sensitivities of the photoreceptors. The theoretical basis of the project is an ideal-discriminator theory (based on the principles of signal detection theory). A computer model of the optics of the eye and the receptor lattice is used to determine the quantum catch in each photoreceptor for the two stimuli in a discrimination task. A mathematically ideal discriminator is then applied to the receptor output. In this way one can determine the physical limits to visual sensitivity imposed by the front-end of the visual system. Psychophysical expeirments measureing acuity, hyperacuity, luminace contrast sensitivity and chromastic contrast sensitivity will then be carried out for comparison with the ideal discriminator's performance. This should allow determination of what aspects of human sensitivity can be explained by properties of the stimuli and the front-end of the visual system up to the receptors. This project should provide a significant step forward in understanding the peripheral visual mechanisms and their contributions to overall visual sensitivity. This is important for understanding normal and abnormal spatial and color vision. The ideal-discriminator theories can also be used to detemrine what information for discrimination is lost by disorders of the optics, receptor lattice, or receptor spectral sensitivities. The project will also make significant contributions toward understanding how to discriminate visual signals in Poisson (quantal) noise. This is a problem of considerable current interest in the area of medical optics--diagnostic imaging devices often produce rather noisy images.

The long range goal of our research program is to develop a quantitative understanding of the mechanisms of visual processing that underlie spatial vision and adaptation in the cone and rod systems. Our primary research strategy has been to work out, in quantitative detail, the constraints on visual performance imposed by anatomical and physiological mechanisms, and to carry out psychophysical tests to determine what aspects of human performance might be accounted for by these mechanisms. The present proposal is to continue this research strategy to the study of spatial-, temporal- and contrast-discrimination performance. One line of research will be to develop and test multiple-channels models of visual adaptation. Much recent research on visual adaptation has involved measuring sensitivity for stimuli with fixed spatial configurations in the flash-on-flash paradigm. This research has been very useful for finding and quantifying the properties of the adaptation mechanism. However, for simplicity, spatial variables have largely been avoided. On the other hand, spatial-vision studies and associated models have largely avoided the transient conditions that most clearly reveal the adaptation mechanisms. Several experiments are proposed to fill this gap and to test models that simultaneously deal with both adaptation and spatial-vision phenomena. These experiments will measure threshold for detection of Gabor patches (of a special type) on transient backgrounds under various adaptation conditions. A second line of proposed research will be to continue the ideal-observer analysis developed in the previous project period, and to extend the analysis to the rod system and to higher levels of visual processing. In the previous project period, we developed an ideal-observer model for the preneural mechanisms up to the level of photon absorption in the cone system, and applied it (with useful results) to a wide range of spatial-, chromatic-and intensity-discrimination tasks. Most of present experiments will be directed at determining what aspects of temporal threshold data might be accounted for by the physical and physiological factors prior to and including the transduction process in the photoreceptors. To analyze the data, we will extend the earlier ideal-observer model to the level of internal-transmitter concentration in the photoreceptor. Finally, a somewhat more exploratory series of studies will attempt to develop an ideal-observer analysis that is appropriate for measuring the information transmitted by single neurons in discrimination tasks. Of particular interest will be the measurement of the information carried in the temporal pattern of the responses.

DESCRIPTION: An established researcher in the area of early vision proposes to use psychophysical and computational methods to study grouping, segregation, and contour integration in multi-element patterns. Specific aims include: 1) the measurement of pairwise trading relationships in grouping strength for symmetry (e.g. how does spatial proximity interact with symmetry in grouping tasks?) 2) comparison of filter vs. structure-based models of grouping and segregation 3) testing of different models of contour integration, 4) measurements of eccentricity effects in texture segregation, 5) modeling of the mechanisms of perceptual organization in perception of complex images. The bulk of the data will be collected with a three element display. The subject's task will be to determine if the center item groups more strongly with the left or right flanking item. In other tasks (e.g. section 4.2) Ss will identify the orientation of a texture patch. A model of grouping is proposed starting with filtering by simple cells as previously modeled by Albrecht and Geisler. A second, "transformational" matching stage allows for similar items to be grouped together. In a complex image, the plan is to have receptive field matching generate a first set of primitives that can be grouped. These groups are then subject to transformational mapping and higher-order groupings can be based on the results of that operation. Grouping is based on a variety of grouping rules (e.g. figure-ground grouping, associative grouping - a principle that holds that if A and B group and B and C group than A and C will group). The model asserts that grouping can be "multi-level" though it is admitted that "our current understanding of multiple-level grouping" is in a rather primitive state.

The long range goal of our research goal has been to develop and test rigorous theories of visual processing that address important aspects of behavioral and neural performance. [I] During the previous grant period we developed a very efficient method (the descriptive function method) for measuring the detection, discrimination and identification performance of single neurons. Briefly, [I] the responses the a neuron are measured along various stimulus dimensions, (2) descriptive functions are fitted to the means and standard deviations of the responses, and then (3) the fitted functions are used to determine single neuron performance. The high efficiency of this method allows us to measure the discrimination (or identification) performance of large populations of neurons and hence compare population performance to behavioral performance. The descriptive function method will be used to determine population performance for a number of stimulus dimensions. [II] Recent work in the primary visual cortex has revealed two important non-linear mechanisms, contrast normalization and response expansion, which make critical contributions to the discrimination and identification performance of cortical neurons. We propose a series of experiments and simulations to characterize the spatial, temporal, and noise properties of these mechanisms. These studies will be important for understanding how the normalization and expansion mechanisms contribute to psychophysical performance, and how they are implemented in the neural circuity of the retina, LGN, and cortex. [III] Quantitative models of using spatiotemporal sinewave granting stimuli. Typically, the stimuli are presented for a fixed duration, in counterbalanced blocks, with careful control of fixation. To test and develop general theories of spatial vision it is important to begin bridging the gap between these carefully controlled stimulus presentation conditions and the more complex stimulus presentations which occur in the natural environment. We propose a series of experiments which will measure neural responses and behavioral discrimination performance for sinewave grating stimuli presented in a fashion which matches the sequence of fixations during saccadic inspection of complex natural images. These data will be compared with results from more conventional presentation methods and will be used to develop quantitative models of spatial visions that are appropriate for natural visual tasks.

Project Abstract

This study is directed towards developing flexible, general-purpose Visual Search systems capable of Searching for objects in real, cluttered environments. The research will include extensive psychophysical and physiological experiments on humans and primates that will prototype artificial systems that mimic this behavior. The goals of the study can be conveniently divided into four Aims:
Aim 1: Develop and prototype a revolutionary camera gaze control device dubbed Remote High-Speed Active Visual Environment, or RHAVEN. RHAVEN will allow telepresent control of the gaze of a remote camera using eye movements as rapidly and naturally as if viewing the scene directly.
Aim 2: Develop optimal statistical bounds on Visual Search, by casting it as a Bayesian problem, yielding a maximum a posteriori (MAP) solutions for firstly, finding a target in a visual scene using a smallest number of fixations, and secondly, for next-fixation selection given a current fixation.
Aim 3: Construct models for Visual Search based on Natural Scene Statistics at the point of gaze. Visually important image structures can be inferred by analyzing the statistics of natural scenes sampled by eye movements and fixations.
Aim 4: Conduct neurophysiological studies on awake, behaving primates during Visual Search tasks. Measure and analyze search performance in awake, behaving monkeys, while measuring the responses of neural populations in the brain's frontal eye fields (FEF) which help control saccadic eye movements.
Broader Impact: The results of this research should significantly impact numerous National Priorities: Searching Large Visual Databases, Robotic Navigation, Security Imaging, Biomedical Search, Visual Neuroscience, and many others. It is easy to envision scenarios that would benefit by a fundamental theory of Visual Search. For example: searching for suspect faces in airport security systems; examining internet streams for questionable material; semi-automatic search for lesions in mammograms; steering robotic vehicles around obstacles in hostile environs; navigating huge visual data libraries, etc.

New, state-of-the-art video quality assessment (VQA) algorithms will be developed that are explicitly designed for use on High Definition (HD) video streams. These will be designed using perceptual criteria and taking into account such human factors as head and eye position. An open database of raw digital HD videos will be developed, along with multiple distorted versions of each video and human subjective scores on the distorted videos. Since HD videos are often resized for display on smaller screen, scalable VQA and IQA algorithms will also be developed that will for the first time, be able to assess the quality of images or videos, in a perceptually significant way that have been scaled or resized from their original dimensions. The development of successful HD Video Quality Assessment (VQA) algorithms that correlate highly with visual perception will represent a major advance in video engineering. The construction of an HD video quality database will be the first of is kind, and certainly heavily accessed by researchers around the world. Prior work by this group on non-HD VQA has resulted in some of the most-highly cited research in the image processing field in the past 20 years. It is anticipated that publications from this work will likewise be highly influential.
The equipment and developed algorithms will also be used as exemplars in the UT-Austin image and video processing educational program. The equipment will be used to generate numerous video processing teaching examples.

How does a vision system recover the 3-dimensional structure of the world -- such as the layout of the environment, surface shape, or object motion -- from the dynamic 2-dimensional images received by the sensors in a camera, or the retinas in our eyes? This problem is fundamental to both computer and biological vision. Computer vision has developed a variety of algorithms for estimating specific aspects of a scene such as the 3-dimensional positions of points whose correspondence over time can be established, but obtaining complete and robust scene representations for complex natural scenes and viewing conditions remains a challenge. Biological vision systems have evolved impressive capabilities that suggest they have detailed and robust representations of the 3-dimensional world, but the neural representations that subserve this are poorly understood and neurophysiological studies thus far have provided little insight into the computational process. This project will pursue an interdisciplinary approach by attempting the understand the universal principles that lie at the heart of 3-dimensional scene analysis.

Specifically, the project will 1) develop a novel class of computational models that recover and represent 3-dimensional scene information, 2) collect high quality video and range data of dynamic natural scenes under a variety of controlled motion conditions, and 3) test the perceptual implications of these models in psychophysical experiments. The computational models will utilize non-linear decomposition - i.e., the ability to explain complex, time-varying images in terms of the non-linear interaction of multiple factors, such as the interaction between observer motion, the 3-dimensional scene layout, and surface patterns. Importantly, the components of these models will be adapted to the statistics of natural motion patterns that arise from observer motion through natural scenes and movement around points of fixation.

The project is a collaboration between three laboratories that have played a leading role in developing theoretical models of natural image statistics, visual neural representations, and perceptual processes. The investigators seek to combine their efforts to develop new models, data sets, and characterizations of 3-dimensional natural scene structure that go beyond previous studies of natural image statistics, and that can be tested in neurophysiological and psychophysical experiments. This project has the potential to bring about fundamental advances in neuroscience, visual perception, and computer vision by developing new classes of models that robustly infer representations of the 3-dimensional natural environment. It will create a set of high quality databases that will be made available to help other investigators study these issues. It will also open up new possibilities for generating realistic stimuli that can guide novel investigations of neural representation and processing.

? DESCRIPTION (provided by applicant): An ultimate goal of vision science is to understand and predict performance under natural conditions, and there is arguably no task more ubiquitous and fundamental than visually searching the environment for objects of interest. However, little is known about search in natural scenes. Our aim is to measure, characterize, and model search performance in natural scenes in both humans and non-human primates. We will study both covert search, where the search scene is presented for a limited time and the eyes remain on a fixation target, and overt search, which involves eye movements. Our approach combines mathematical/computational analysis, psychophysical and eye-movement measurements in humans and monkeys, and voltage sensitive dye imaging (VSDI) in V1 of behaving monkeys. We argue that a rigorous understanding of visual search must begin with a characterization of the sensory factors that control the detectability of the target across the visual field in absence of any uncertainty about the location of the target. In preliminary psychophysical, neurophysiological, and computational studies we developed a model of target detectability that predicts simple detection performance in natural images. In further computational studies (based on the detection results), we discovered a biologically-plausible computation that would allow the brain to perform optimal overt search in natural scenes. Specifically, we prove that during a visual search, the optimal next fixation location is obtained from the current belief map (posterior probability map) of where the target might be located, by (i) dividing (normalizing) the map by the local image contrast, (ii) blurring this contrast-normalized map appropriately, and then (iii) selecting the peak of the blurred map as the next fixation location. The proposed studies will measure human and monkey detection and visual search performance in natural images, and will measure VSDI responses in monkeys during fixation and detection tasks. The studies will let us test and further develop our models of simple detection and covert search in natural images, and determine which components of optimal fixation search humans use when searching in natural images. They may also lead to methods for training humans to use more optimal search strategies.

The overarching goal of this proposal is to develop an optical-genetic toolbox for reading and writing neural population codes in functional maps of awake, higher mammals. Such tools could ultimately be used to restore perceptual capabilities in patients with damage to peripheral sensory pathways by direct stimulation of early sensory cortex. Advanced optical methods for reading and writing neural codes using genetically-encoded reporters and actuators have become powerful tools for studying neural circuits in rodents. However, rodents are a suboptimal model for human perception because of their vastly different sensory representations and perceptual capabilities. For example, rodents' primary visual cortex (V1) lacks the functional columnar organization which is a hallmark of primate vision. In contrast to rodents, the macaque monkeys' sensory representations and perceptual capabilities are highly similar to those of humans. Furthermore, the behaving macaque provides a unique opportunity to develop and test tools for reading and writing neural codes at the level of functional domains such as the orientation columns in V1. However, multiple technical hurdles remain before the optical-genetic methods currently available in rodents could be readily applied in larger, non- transgenic mammals. Here we propose to take advantage of the unique expertise of our team members to develop optical techniques that utilize virally delivered transgenes for monitoring and manipulating neural population codes in behaving macaques. Specifically, we will address three technical goals. First, we will develop and test new genetic methods that will provide long-term expression of transgenes in primates with cell-type and activity- dependent specificity. Second, we will develop a two-photon microscope for behaving monkeys that will allow one to monitor these signals with cellular resolution and complement current imaging techniques with larger coverage but coarser resolution. Finally, we will develop methods for writing neural population codes in functional maps by combining patterned light stimulation that target specific functional domains and selective expression of actuators. We will validate and optimize these techniques by linking V1 responses (elicited by both visual and direct patterned optogenetic stimulation) and monkeys' behavior in visual discrimination tasks. The tools that we will develop will enable a deeper understanding of the neural code and a better characterization of the capabilities and limitations of methods for reading and writing neural population codes in functional maps in humans.

Project Summary/Abstract The Center for Perceptual Systems (CPS) is an interdisciplinary program at the University of Texas that provides a focal point for research and training in sensory systems. Continued growth in the Neurosciences at UT, as well as in CPS, particularly in the area of vision research, has lead to the development of a distinctive group in the Center that has a broad interdisciplinary research program in vision, representing psychophysical, neurophysiological, imaging, and computational approaches. Additionally, we have distinctive expertise and resources for the investigation of vision in the context of natural environments, and the development of computational tools for neural modeling and for modeling visually guided behavior. Because of both our broad expertise and our special strengths in these areas we are well positioned to train new scientists who will be at the forefront of vision research in the future. There are three important components to our training focus. First, we take advantage of the highly interdisciplinary and collaborative structure of CPS to provide broad cross- disciplinary training, which we consider essential for students of vision and visual performance. Second, we believe that training in computational methods is an essential component of research in vision and we take advantage of our particular strengths in this area. Third, we take advantage of our strengths in the area of natural systems analysis to provide a unique training opportunity in an area that is becoming increasingly important for a broad understanding of vision. These components lie at the core of our program, which includes basic courses, specialized seminars, training in advanced methodologies, attendance at the CPS colloquium series and the CPS symposium on Natural Environments, Tasks, and Intelligence, ethics training, and the development of professional skills.

Project Summary/Abstract Visual systems must be matched (via evolution and learning over the lifespan) to the natural tasks organisms perform to survive and reproduce. Thus, it is of fundamental importance to analyze visual systems with respect to natural tasks and with respect to the statistical properties of natural stimuli relevant to performing those tasks. In our lab we call this ?natural systems analysis.? This novel approach to vision science is composed of several steps: (1) identify natural tasks, (2) measure the natural scene statistics relevant for those tasks, (3) determine how to optimally use those statistics to perform the tasks, given appropriate biological constraints, and (4) use the first three steps to formulate principled hypotheses which are tested and refined in behavioral or physiological experiments. Using a unique suite of measurement devices, computational tools, and psychophysical paradigms developed in our laboratory, we propose to tackle (within the framework of natural systems analysis) several fundamental tasks involving estimation of local properties in natural scenes: (Aim 1) detection of occluding and partially-occluded targets in natural images, (Aim 2) detection of depth edges created by occluding surfaces and estimation local 3D surface orientation at the non-depth edge locations within those surfaces, and (Aim 3) estimation of disparity and local 2D motion. Many of the proposed studies will be the first to precisely characterize the statistical constraints in natural images underlying the visual system's ability to perform these tasks accurately. Many of the proposed studies will also be the first to measure performance in these fundamental tasks using natural stimuli. The product of the studies will be not only unique new measurements, but principled new models that can predict human performance under natural conditions and guide future neurophysiological studies of the underlying mechanisms. Strong preliminary results have been obtained in the previous project period for many of the proposed studies.