Yali Amit - US grants

Abstract

PI: Donald J Geman

Title: ITR SMALL: Invariant Detection and Interpretation of Specific Objects in Image Data

The area of investigation is automated scene analysis. The main objective is to detect the appearance in image data of objects from a small repertoire. Two key liabilities in current methods are insufficient invariance, both photometric and geometric, and inefficient computation. To confront these difficulties, a unified statistical and computational framework is proposed which is based on a coarse-to-fine sequence of approximations to a full Bayesian model. Research topics include both algorithmic and mathematical issues arising in coarse-to-fine search, model selection and deformable shape analysis.

The interpretation of natural scenes is effortless for human beings but is the main challenge of artificial vision. This "semantic gap" has largely resisted any satisfying solution and impedes scientific and technological advances in many areas, including automated medical diagnosis, industrial automation, and effective security and surveillance. The general objective of this project is to design computer algorithms for detecting and interpreting certain objects appearing in still pictures in order to relieve humans of wearisome visual search tasks in medical imaging, law enforcement, industrial inspection and everyday life

Proposal: 0427223
Principal Investigators: Donald Geman, Yali Amit, Stuart Geman, and Laurent Younes
Institutions: Johns Hopkins U, U of Chicago, Brown U, and Johns Hopkins U
Proposal Title: ITR-\(ASE+NHS)\-\(dmc+int)\: Triage and Automated Annotation of large Image Data Sets

ABSTRACT

The long-term goal is a computational and mathematical framework for progressive annotation of large image databases at a semantic level. The proposed research fuses two powerful paradigms, coarse-to-fine indexing and syntactic scene parsing, into one model and computational strategy in order to provide a less-to-more detailed annotation of a scene as a function of available computing cycles. Coarse, and likely flawed, interpretations emerge at the early stages of processing, followed by finer and more accurate ones. The fusion between coarse-to-fine and syntactic models allows one to gain the best of both: a nearly optimal computational engine for detecting candidate constituents embedded in a full semantic and syntactic scene analysis.

Large image data sets are ubiquitous. Sources include medicine, manufacturing, astrophysics, molecular biology, defense and intelligence. In general, these resources are useful only in proportion to our ability to access selected semantic categories, such as ``includes people'' or ``contains a frontal lobe mass''. There is then a great need for automated annotation, whereby computer programs would produce a ``meta'' data structure, partially describing the contents and context of each image. Progress in automated scene annotation will have an immediate impact on a broad range of scientific disciplines, including medicine and surveillance .

The introduction of statistical techniques in computer vision has yielded a number of interesting algorithms able to partially solve certain constrained recognition problems. However limitations on computing power and available training data impose certain difficult tradeoffs which are rarely quantified so that choices of parameters and models are typically done in an ad-hoc manner. These tradeoffs can only be quantified in a context where the statistical properties of the objects and their appearance in the images are well defined, yet this is far from the case in real images. The alternative, which is the goal of this project, is to perform an analysis of the same issues in a synthetic stochastic setting, using a generative model for images. Object classes are stochastically generated and instantiated in the images, together with clutter, occlusion and noise. The generative model should be rich enough to qualitatively pose the same problems as real images, yet sufficiently simple to enable quantitative analysis; hence this is not an attempt to synthesize real images. Questions regarding the limits of feasibility of various tasks such as detection and classification as a function of key parameters defining the generative model is analyzed quantitatively, in particular the analysis of the tradeoff between accuracy and computation time. The emphasis on integrating computation time in the analysis gives rise to new types of statistical questions, and new forms of asymptotic regimes as a function of the image resolution, the number of distinct classes and their variability. The hope is that the proposed framework will offer a setting in which systematic algorithmic choices can be made and contribute to the development of concrete computer vision algorithms.

Computer vision algorithms have produced some partial solutions to some constrained problems such as face detection, hand written digit recognition, or face recognition in severely restricted settings. Since a proper theoretical foundation for the field is lacking, a wide variety of algorithms have been proposed based on ad-hoc choices and it is difficult to assess what components of the different approaches are the most useful, which elements should be extended further and which elements should be dropped. The first step in laying a theoretical foundation for computer vision algorithms is a statistical description of the population of images. Since this is very hard to define the investigators propose to study a synthetically generated world of images, which is much simpler but which gives rise to qualitatively similar tradeoffs and challenges. In this synthetic setting the investigators will rigorously quantify the tradeoffs and hopefully be able to draw important conclusions with respect the algorithmic applications.

This proposal aims to integrate two streams of research on learning and memory in an attempt to strengthen the links between theory and experiment, build models that explain experimental observations and use model predictions to guide new experiments. The experimental stream will record neuronal population activity in inferior temporal, perirhinal and prefrontal cortices during performance of delayed matching tasks which require maintenance of visual information in short term memory, using visual stimuli with various degrees of familiarity (from entirely novel to highly familiar). The modeling stream will investigate learning and memory in network models that include learning rules inferred from data, using a combination of mean field analysis and simulation. Models will generate predictions on patterns of delay period activity that will be tested using experimental data. The goals of this combined experimental and theoretical project will be to answer the following questions: · How do changes in synaptic connectivity induced by learning due to repeated presentation of a particular stimulus affect the distributions of visual responses of neurons? In other words, how do neuronal representations change in cortex as a novel stimulus becomes familiar? Can we infer the learning rule in cortical circuits from experimentally observed changes in distributions of neuronal responses as the stimuli become familiar? · Do changes in synaptic connectivity induced by learning rules that are consistent with the statistics of visual responses lead to delay period activity in a task such as the OMS task? Is delay period activity already present upon the first presentation of a stimulus, or does it develop over time? If it is not present during the initial presentations, how is sample information maintained in memory during the delayed match to sample task? see attached continuation RELEVANCE (See instructions): See attached