Marshall Tappen - US grants

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5). The project represents a new paradigm for research in computer vision related to content-based image management--?one that exploits a symbiosis between this technology and the online communities it serves. It is quite clear that communities stand to benefit from improved searching and browsing tools enabled by visual technology, but we argue that the converse is equally true. Visual information that is shared online is almost always embedded in a social network, and these networks provide rich contextual information that can be a boon to vision systems.

Our goal is to build foundations for a new generation of computer vision systems that are 'socially aware'. These vision systems will exploit knowledge of the relationships between individuals and the social events that occur in the embedding online community, and in doing so they will exhibit significant performance gains over current technology. They will be trained discriminatively using the data embedded in existing online communities, and they will adapt in response to observations of how they are used by community members.

Almost every image that is shared online is embedded in some form of online community, and any system that seeks to analyze and catalog such imagery will gain from the proposed research. We foresee benefits for communities that wish to share historical image archives, scientific image collections, forensic imagery, surveillance videos, and more. Also, while we focus on tools for searching and browsing imagery, our work will have the important effect of improving the quality and density of textual metadata. In this way, the research stands to benefit all members of an online community, including those with visual impairments.

This project focuses on developing algorithms and datasets that can transform photometric reconstruction systems from hand-designed systems into learning-based systems that are optimized on real-world data.
Photometric reconstruction systems derive cues from the perceived intensity of different locations on a surface. Shape-from-shading, where the surface is assumed to have a diffuse reflectance, is a well-known example of photometric reconstruction. This project produces the datasets and methods necessary to use machine learning techniques to build models for photometric reconstruction.

This learning-based approach enables systems to be optimized on real-world data so that they produce the most accurate results possible. In addition, this learning-based approach enables the development of more sophisticated methods with more parameters than typically used in hand-designed systems. The ability to find optimal parameters in an automated fashion can not only improve existing approaches, such as by incorporating image data more effectively, but can also enable the development of algorithms that push the boundaries of current systems. In particular, algorithms are developed for estimating the shape of objects without knowing the illumination or even trying to explicitly model it.

The power of the learning approach cannot be realized without data for training and testing. A major task in this work is the construction of a database of images and ground-truth 3D reconstructions of the objects in the images. The 3D models can be found using an example-based photometric stereo technique.

This project is creating a novel paradigm for computer vision, termed "reconstructive recognition", that incorporates the strongest elements of previous machine learning-based recognition efforts and the strongest elements of previous reconstruction efforts based on radiometric reasoning. The goal is to provide a new foundation for machine perception, and the potential for a transformative advance in applications of computer vision. The project seeks novel physics-based methods for recognition as well as novel learning-based methods for interpreting pixel values in terms of the physics of a scene. The agenda is structured around four aims: Aim I develops generalized reconstructive processes that unify the recovery of shape, materials, motion and illumination. Aim II focuses on supervised visual learning methods that exploit such reconstructive image representations. Aim III pursues unsupervised discovery of reconstructive representations that converge to be similar to the engineered models of Aim I. Finally, Aim IV introduces well-defined challenge problems that focus the field and serve as measurable proxies for progress in computer vision applications that have high potential impact on society.

There is a significant broader impact to this project, not least being the improvement in computer vision pedagogy that ensues from a reunification of the currently divergent recognition and reconstruction views of the field. More broadly, this project pursues critical steps toward a future where machines can see, a future that will bring changes to robotics, human-computer interfaces, security, and autonomous navigation, to name a few.