David William Jacobs - US grants

Shape analysis is a fundamental problem in computer vision. Object recognition requires comparison of two shapes and determination of similarity. The challenge is that an object?s shape may vary due to articulations, non-rigid deformations, or natural variations between different instances of objects from the same class. It is important to understand how to represent shape, how to match shape allowing for deformation, and how to learn the types of variations that can occur for a particular class of objec5ts. The PI has a number of new approaches to shape matching and has been applying them to the construction of devices for plant species identification. The system can be used to monitor distribution or discover new species. Shape analysis has a wide range of additional applications, from military ones to medical ones. The PI will use the funds to develop new collaborations in statistical models of shape.

This project is to develop new, effective distance metrics for comparing two images. These metrics account for two effects. First, pixels can change their position, deforming from one image to another. Second, pixels may change their intensity. In many vision problems, intensity changes are primarily due to lighting variation. The research team first addresses the effect of illumination changes, which enables to develop a new, powerful, robust distance for measuring the effects of lighting variation in an image. The research team combines this with both existing and new methods to develop a robust distance that accounts simultaneously for image deformations and intensity variations. Computing this distance separates these two effects, providing a correspondence between images. This can be used to track objects moving relative to a light, to match images taken at different times of day, or to recognize objects seen under different lighting, from different viewpoints, with variations in their shape.

This new metric provides a theory of computation for deformation and lighting that encodes our notion of image similarity. However, it is still a considerable challenge to find ways to effectively compute with such an image metric. Therefore, the research team also develops computationally effective algorithms based on this new metric. These algorithms improve performance in numerous applications such as face recognition, autonomous navigation, and optical flow and tracking, in which variations in lighting and shape cause significant challenges for existing methods.

Introduction
This project will make use of a novel system that will serve to accelerate the documentation of millions of currently unknown species. It will involve the use of mobile phones and the internet, which together make it feasible for thousands (if not millions) of people to collaborate on large-scale projects of tremendous social importance. In this project, data on biodiversity will be collected by citizen-scientists using mobile phones. There are approximately two million known species of organisms in the world and potentially millions more are still undocumented. Without help, professional biologists will be unable to faithfully record many of these species before they disappear from the planet.

Intellectual Merit
The research goal of this project is to develop and test evolving theories for designing socially intelligent systems in which enthusiasts (usually referred to as "citizen-scientists") and scientist?s partner with technology to collect and process the data needed for large scale observation-driven science. This study will extend existing work by examining motivations of expert scientists and novice enthusiasts partnered with computing technologies in a scientific domain; developing and testing novel theory-inspired strategies for motivating participants; and beginning to develop a meta-theory that characterizes the interplay between potentially competing motivational strategies or "design levers" in the same socially intelligent system.

Potential Broader Impacts
Inventorying and compiling basic descriptions of a fraction of the world's species is a huge task that cannot be accomplished by trained scientists alone. This project will help enlist the public to accomplish this task. The Encyclopedia of Life project will employ the techniques that are developed in this project for use by other volunteers to obtain additional information about species around the globe.

The automatic identification in images of people, places, objects, and especially object categories is a central and ongoing challenge within computer vision. This project addresses this problem using low-level image features to learn intermediate representations, ones in which objects in images are labeled with an extensive list of highly descriptive visual attributes. This work demonstrates this approach in three domains: faces, plant species, and architecture. In each domain, it develops techniques for deriving visual attribute vocabularies, training attribute detectors, and building compositional models to automatically label attributes in images.

The project is making four fundamental contributions to the use of visual attributes. 1) It is developing new methods by which automatic systems and humans can interact to select domain-appropriate attribute vocabularies and label large image collections. 2) It is developing compositional models that capture dependencies between attributes. This provides more accurate attribute detection and enables inference of global properties of objects. 3) Using compositional models, the project is developing new, localizable attributes that capture the geometric relations between object parts and landmarks. 4) The project is designing algorithms that combine attributes to identify objects, search through image vast collections, and automatically annotate image databases.

Not only is this research generating large datasets of labeled images that should help catalyze new research, it is also demonstrating the feasibility of new systems for analyzing images in specialized domains such as faces, plants, and architecture. For example, the project develops new software applications for analyzing and searching images of faces as well as free mobile apps for plant species identification.

Despite comprehensive civil rights legislation for Americans with disabilities, many city streets, sidewalks, and businesses remain inaccessible. The problem is not just that street-level accessibility affects where and how people travel in cities but also that there are few, if any, mechanisms to determine accessible areas of a city a priori. Traditionally, sidewalk assessment has been conducted via in-person street audits, which are labor intensive and costly, or via citizen call-in reports, which are done on a reactive basis. And while efforts exist for visualizing the walk-ability, bike-ability, and availability of public transport in cities, there are no analogous efforts for accessibility. Thus, wheelchair users, for example, often avoid going to new areas of a city where they don't know about accessible routes. The PI plans to address this problem by means of a two-pronged approach in which he will first develop scalable data collection methods for acquiring sidewalk accessibility information using a combination of crowd-sourcing, computer vision, and online map imagery; he will then use the new data to develop and evaluate a novel set of navigation and map tools for accessibility. To these ends, the PI and his team will collect and analyze interview and survey data both from mobility impaired persons and from ADA streetscape design experts, and will seek to understand how people with mobility impairments can make use of interactive mapping information to enhance mobility. They will study methods for efficiently and effectively crowd-sourcing map labeling tasks, evaluating existing approaches empirically and designing novel, more effective approaches. They will develop new computer vision algorithms for the analysis of street scenes, which will be used to help scale the data collection by focusing human labeling efforts on locations that are most likely to contain significant problems. And they will design, implement and evaluate new accessible-aware map-based tools to aid people with mobility impairments in navigating their cities. As appropriate for each phase of the research, user evaluations will include both lab and field studies.

Broader Impacts: Roughly 30.6 million individuals in the United States have physical disabilities that affect their ambulatory activities, and nearly half of these individuals report using an assistive aid such as a wheelchair, cane, crutches, or walker. The outcomes from this research will have a significant impact on the ability of these Americans to travel independently, by transforming the ways in which accessibility information is collected and visualized for every sidewalk, street, and building façade in America. Project outcomes will include a publicly accessible web site where both the labeled data collected during this work and the new prototype tools developed will be made available for general use. Furthermore, the PI and Co-PI will advise and mentor both graduate and undergraduate students throughout the course of the project, including two PhDs and two MS students who will obtain a cross-disciplinary education in human-computer interaction and computer vision.

This project develops technologies for understanding shapes and parts of a person or animal and how these relate to their surface appearance. By building models that capture the variations in shape and pose of humans and animals, it becomes possible to understand the way that a person or animal's appearance changes as its arms and legs move or its head turns. These models can then be aligned with images, assisting in the recognition of figures and the determination of their pose. Understanding human pose and activity is a fundamental problem in computer vision with a host of interesting applications in surveillance, video retrieval, and automated video annotation. Automated systems that can identify the species of animals can form the basis for automated field guides that can be used in education and studies of biodiversity.

This research develops new algorithms for matching image features and registering 3D models with bounded distortion mappings. The research team models people and animals using a skeleton capturing their articulations, along with a deformable skin model and an appearance model encoded by classifiers that can identify body parts of an animal. Given an image, the system computes an optimal bounded distortion transformation to register the model with the image. The system identifies both the pose and shape change of the person or animal with respect to the model and provides a way to rank possible detections of the model. The research team explores the problem of identifying the species of animals. The research team further applies algorithms to determine the pose of humans in images.

Deep neural networks have had a huge recent impact on the world. They are widely used in systems that understand speech, translate language, and analyze images. In spite of their great impact, researchers still lack a rigorous understanding of many of the basic properties of these networks. As a consequence, new networks are largely designed laboriously, through trial and error. And although extremely effective overall, these systems are sometimes fooled by examples that seem very simple, and similar to other examples that are easily handled. This research aims to provide a better theoretical understanding of an important class of neural networks, called Convolutional Neural Networks (CNNs), which are widely used in understanding images and audio signals. The project focuses on understanding what problems will be easy or difficult for CNNs. This understanding can help us to predict biases in these networks and understand how the design of a network will affect its behavior. The project will provide research opportunities for graduate, undergraduate and high school students, particularly reaching out to students from underrepresented groups.<br/><br/>Two key properties that distinguish CNNs from many other approaches to machine learning are their ability to naturally incorporate multiscale analysis and invariance or equivariance. This property has enabled the construction of shift invariant networks that effectively deal with images and signals sampled on grid data, and more recently of networks that handle sets and graphs, incorporating operations that are equivariant to set permutation and graph isomorphism. Multiscale representations naturally arise in these networks through their depth. This research focuses on gaining a better understanding of the role of multiscale, invariance and equivariance in neural networks. It will study how shift invariance and multiscale representations affect the dynamics of neural network training. Our approach will build on recent results showing that massively overparameterized neural networks can be represented as kernel methods. Analyzing the properties of these kernels will help us understand the relationship between a network's architecture and its inductive biases. The insights revealed have the potential to provide a more principled way to control these biases.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.