Lawrence K. Cormack - US grants

The proposed project involves both psychophysical and computational investigations of human stereopsis and binocular vision. To date, most computional investigations have focused on solving the binocular matching problems while most of the psychophysics have focused on stereoscopic localization or stereoscopic form recognition. The broad goal of this project is to gain a greater understanding of the process of stereopsis by alleviating this imbalance. The specific aims of the proposed project are: 1) To investigate the psychophysical properties of the binocular matching process in human vision and to compare and contrast these properties with those of stereoscopic localization. 2) To explore the nature of disparity channeling in the human visual system. This includes: a) Developing computational models based on both the classical 3-channel theories of stereopsis and more recent theories based on larger number of channels. b)Testing both of the above types of models psychophysical using two types of tasks, those that involve binocular matching and those that involve binocular matching and those that involve stereoscopic localization. c) Reassessing the meaning of the clinical conditions of stereoblindness and stereoanomaly in light of the results of the above investigations. 3) To develop a computational model of stereopsis that incorporates a biologically plausible front end and that can make quantitative predictions about two types of psychophysical experiments, those involving binocular matching (e.g. correlation detection) and those involving stereoscopic localization (e.g. stereoacuity). The psychophysical experiments will involve one of two kinds of tasks. In the first type of task, observers will be required to make judgments concerning the interoccular correlation of dynamic random element stereograms, which will be generated and displayed on a microcomputer. No judgments concerning disparity or depth will be required. In the second type of task, observers will be required to make judgments involving stereoscopic localization under nearly identical stimulus conditions. The differential effects of various stimulus variables on these two types of judgments will be analyzed to distinguish the properties of stereopsis that are governed by the mechanisms of binocular matching from those that are governed by the mechanisms of stereoscopic localization.

ABSTRACT

"ACTIVE STEREOSCOPIC VISUAL SEARCH DRIVEN BY NATURAL SCENE STATISTICS"

Alan C. Bovik, Lawrence K. Cormack, J. Ghosh

The primary thrust of this proposal is to develop methods based on the natural statistics of stereoscopic images that will enable the design and implementation of the next generation of foveated, fixating machine vision systems that are capable of efficient and intelligent visual search, by exploiting and applying knowledge about human fixation and search mechanisms.
We summarize the intention of our proposal via the following key goals:
Goal 1: To develop a quantitative description of human active stereo vision as a function of natural scene statistics in a variety of three-dimensional visual search and learning tasks. Our emphasis will be on developing statistical models of stereo primitives that attract low-level visual attention based on a unique and in-depth statistical analysis. We feel that statistical models based on natural scene statistics have a very good chance of succeeding where deterministic models have failed.
Goal 2: To train a state-of-the-art foveated, fixating active computer vision system (named FOVEA) to search and to learn to search 4-D (space-time) scenes. To do this, back-end artificial neural networks trained on telepresent human search patterns will be used. The statistical models and extracted statistical stereoprimitives discovered as part of the research in Goal 1 will be used as a priori knowledge to improve the configuration and learning of the networks. We envision that these experiments will result in smart active machine vision protocols for exploring, searching, and interacting with 4-D environments, while giving new insights into visual cognitive processes.

Project Abstract

This study is directed towards developing flexible, general-purpose Visual Search systems capable of Searching for objects in real, cluttered environments. The research will include extensive psychophysical and physiological experiments on humans and primates that will prototype artificial systems that mimic this behavior. The goals of the study can be conveniently divided into four Aims:
Aim 1: Develop and prototype a revolutionary camera gaze control device dubbed Remote High-Speed Active Visual Environment, or RHAVEN. RHAVEN will allow telepresent control of the gaze of a remote camera using eye movements as rapidly and naturally as if viewing the scene directly.
Aim 2: Develop optimal statistical bounds on Visual Search, by casting it as a Bayesian problem, yielding a maximum a posteriori (MAP) solutions for firstly, finding a target in a visual scene using a smallest number of fixations, and secondly, for next-fixation selection given a current fixation.
Aim 3: Construct models for Visual Search based on Natural Scene Statistics at the point of gaze. Visually important image structures can be inferred by analyzing the statistics of natural scenes sampled by eye movements and fixations.
Aim 4: Conduct neurophysiological studies on awake, behaving primates during Visual Search tasks. Measure and analyze search performance in awake, behaving monkeys, while measuring the responses of neural populations in the brain's frontal eye fields (FEF) which help control saccadic eye movements.
Broader Impact: The results of this research should significantly impact numerous National Priorities: Searching Large Visual Databases, Robotic Navigation, Security Imaging, Biomedical Search, Visual Neuroscience, and many others. It is easy to envision scenarios that would benefit by a fundamental theory of Visual Search. For example: searching for suspect faces in airport security systems; examining internet streams for questionable material; semi-automatic search for lesions in mammograms; steering robotic vehicles around obstacles in hostile environs; navigating huge visual data libraries, etc.

New, state-of-the-art video quality assessment (VQA) algorithms will be developed that are explicitly designed for use on High Definition (HD) video streams. These will be designed using perceptual criteria and taking into account such human factors as head and eye position. An open database of raw digital HD videos will be developed, along with multiple distorted versions of each video and human subjective scores on the distorted videos. Since HD videos are often resized for display on smaller screen, scalable VQA and IQA algorithms will also be developed that will for the first time, be able to assess the quality of images or videos, in a perceptually significant way that have been scaled or resized from their original dimensions. The development of successful HD Video Quality Assessment (VQA) algorithms that correlate highly with visual perception will represent a major advance in video engineering. The construction of an HD video quality database will be the first of is kind, and certainly heavily accessed by researchers around the world. Prior work by this group on non-HD VQA has resulted in some of the most-highly cited research in the image processing field in the past 20 years. It is anticipated that publications from this work will likewise be highly influential.
The equipment and developed algorithms will also be used as exemplars in the UT-Austin image and video processing educational program. The equipment will be used to generate numerous video processing teaching examples.

This project investigates two deeply commingled and significant scientific questions on the statistical distributions of range, disparity, chrominance and luminance in natural 3D images of the world: (1) developing a comprehensive database of co-registered luminance, chrominance, range, and disparity images of natural scenes; and (2) conducting eye movement studies on stereoscopic images.. On the acquired database, the research team studies and models the bivariate statistics of luminance, chrominance, range, and disparity . In the eye movement studies, the locations of visual fixations are measured as they land in range space against where they land in luminance, chromatic, and disparity space, making it possible to develop gaze-contingent models of the statistics of luminance, chrominance, range, and disparity. The results of these studies have broad significance in vision science and image processing. To exemplify this, new approaches to computational stereo and to stereo image quality assessment are developed. New computational stereo algorithms are developed using appropriate prior and posterior distribution models on disparity. Further, new algorithms are developed for stereopair image quality assessment using the statistical models that we will develop. These new algorithms dramatically impact the emerging 3-D digital cinema, gaming, and television industries, allowing for automatic assessment of 3D presentations to human viewers. The developed 3D range-luminance databases are made available via public web portals, and the results of the work are published in the highest-profile vision science and image science journals.

DESCRIPTION (provided by applicant): Objects move through the environment in three dimensions, and humans are clearly capable of perceiving such motion in depth. Visual neuroscientists do not, however, know how our nervous system encodes this fundamental aspect of the visual world. Despite large literatures motion and depth perception, there is a surprising lack of knowledge integrating these two visual features to directly characterize how the brain processes the three-dimensional direction of moving objects. The goal of this proposed research is to understand how neural circuits in the primate brain exploit binocular information to represent the direction of objects moving through a 3D environment. First, we will psychophysically characterize the binocular cues to 3D motion. Recent work suggests that the perception of motion through depth relies on two binocular cues, one based on changing disparities over time, and one based on an inter-ocular comparison of velocities. Our overarching hypothesis is that the velocity-based cue is of great importance for the perception of 3D motion. We will therefore perform psychophysical experiments that isolate eye-specific motion signals and characterize them relative to (and in interaction with) disparity-based signals. These experiments will unpack the psychophysical building blocks and signatures of this important perceptual information, and refine visual displays used for neuroimaging and electrophysiological studies. Second, we will use neuroimaging to identify the neural circuits that process 3D motion. Taking stimuli and insights from our psychophysical experiments, we will perform fMRI experiments to visualize this processing in the human brain. Direction-selective adaptation protocols will be used to characterize both the disparity-based and velocity-based cues and their interactions, and to understand how (or if) these cues are integrated into a single (cue-independent) representation of 3D motion. These experiments will also assess how directly psychophysical assays of the system map on to neural signals. Third, we will perform electrophysiology to specify the underlying neural computations. Guided by the psychophysics and neuroimaging, we will perform single-neuron recordings to characterize the binocular neural signals that encode 3D motion. Recordings in V1 will employ multi-electrode arrays; recordings in MT will employ both multiple-tetrode and single-electrode awake preparations. This work will test the hypothesis that eye-specific motion signals are represented at the level of V1 and are then integrated (by single neurons) in MT. This 3D motion pathway may be exposed at slower speeds, and thus may be multiplexed in the same circuitry known to extract 2D/frontoparallel direction when assessed using faster speeds.