Kristen Grauman, Ph.D. - US grants

Abstract

Title: Scalable Image Search and Recognition: Learning to Efficiently Leverage Incomplete Information
PI: Kristen Grauman
Institution: The University of Texas at Austin


As it becomes increasingly feasible to capture, transmit, and store image and video content on a large scale, the need for machine vision algorithms capable of interpreting it is undeniable. The opportunities appear vast, but progress towards large-scale visual recognition hinges on the development of computationally efficient methods that can effectively leverage minimal supervision. The proposed research considers how informative but incomplete cues can contribute to the learning process, with the goal of enabling large volumes of visual data to be efficiently organized and queried, and a greater number of visual categories to be recognized.

This project intends to advance the scale of the recognition problem by using fragments of supervision, even when they are inexact or dynamic. The PI and her team will develop methods to allow very large image databases to be searched according to distance functions inferred from sparse similarity constraints. They will consider visual category learning scenarios where the system itself actively requests only the most useful information, and integrates ambiguous cues from external modalities such as text. As knowledge about an image collection evolves over time, so must the associated search structure. The PI will investigate ways to adapt image indexing techniques according to dynamic constraints. The proposed technical plan calls for a combination of ideas from vision, learning, and algorithms. Scalable recognition and image search will affect the extent to which visual data can be accessed and mined, making this work relevant to other scientific disciplines where images capture vital information but currently lack proper tools for large-scale analysis. The project also entails complementary educational and outreach activities aimed at engaging students in research, furthering communication across related areas, and encouraging young students to consider studying computer science or engineering.

Updates will be available from: http://www.cs.utexas.edu/¡­grauman/

This project explores (semi-)automatic ways to create "semantically discriminative" mid-level cues for visual object categorization, by introducing external knowledge of object properties into the statistical learning procedures that learn to distinguish them. In particular, the PIs investigate four key ideas: (1) exploiting taxonomies over object categories to inform feature selection algorithms such that they home in on the most abstract description for a given granularity of label predictions; (2) leveraging inter-object relationships conveyed by the same taxonomies to guide context learning, so that it captures more than simple data-driven co-occurrences; (3) exploring the utility of visual attributes drawn from natural language, both as auxiliary learning problems to bias models for object categorization, as well as ordinal properties that must be teased out using non-traditional human supervision strategies; (4) mining attributes that are both distinctive and human-nameable, moving beyond manually constructed semantics.

The project entails original contributions in both computer vision and machine learning, and is an integral step towards semantically-grounded object categorization. Whereas mainstream approaches reduce human knowledge to mere category labels on exemplars, this work leverages semantically rich knowledge more deeply and earlier in the learning pipeline. The approach results in vision systems that are less prone to overfit incidental visual patterns, and representations that are readily extendible to novel visual learning tasks. Beyond the research community, the work has broader impact through inter-disciplinary training of graduate and undergraduate students, and outreach to pre-college educators and students through workshops and summer camps encouraging young students to pursue science and engineering.

The goal of this proposed infrastructure is to enable research towards the creation of the first buildingwide, pervasive, life-long learning, multi-robot intelligence that is situated within an existing building. This buildingwide intelligence (BWI) infrastructure will be comprised of multiple robot appendages, as well as a suite of static sensors and interactive displays. The ultimate goal of the facilitated research is to enable BWI to interact naturally with all of the buildings users, both learning about the patterns and preferences of its long-term inhabitants, and aiding first-time visitors. In addition, it will be robust to continual changes to the capabilities and availability of its component robots and other sensing and actuation resources; and it will consist of readily available components and open source software for ease of replication.

The most important broader impact of this proposal will be the development and dissemination of a new, open infrastructure suitable for cutting-edge research across the department and university, including reusable components suitable for installation in buildings everywhere. The concept of BWI will also provide impacts on education, including a new curriculum for a project-based course, serving as a platform for graduate and undergraduate research (including as a part of UTs novel Freshman Research Initiative), as well as a very public facade, which will enable public demonstrations at outreach events targeting underrepresented groups such as women in CS and Hispanics.

Today there is far more video being captured - by consumers, scientists, defense analysts, and others - than can ever be watched. With this explosion of video data comes a pressing need to develop automatic video summarization algorithms. Video summarization takes a long video as input and produces a short video as output, while preserving its information content as much as possible. As such, summarization techniques have great potential to make large video collections substantially more efficient to browse, search, disseminate, and facilitate communication. Such increased efficiency will play a vital role in many important application areas. For example, with reliable summarization systems, a primatologist gathering long videos of her animal subjects could quickly browse a week's worth of their activity before deciding where to inspect the data most closely. A young student searching YouTube to learn about Yellowstone National Park could see at a glance what content exists, much better than today's simple thumbnail images can depict. An intelligence agent could rapidly sift through reams of aerial video, reducing the resources required to analyze surveillance data to identify suspicious activities.

This project develops new machine learning and computer vision algorithms for video summarization. Unsupervised methods, which are the cornerstone of nearly all existing approaches, have become increasingly limiting due to their reliance on hand-crafted heuristics. By instead posing video summarization as a supervised learning problem, this project investigates a markedly different formulation of the task. The research team is investigating four key new ideas: powerful probabilistic models for learning to select the optimal subset of video frames for summarization, semi-supervised learning models and co-summarization algorithms for leveraging the abundance of multiple related videos, algorithms for exploiting photos on the Web to improve summarization, and evaluation protocols that assess summaries in a way that aligns well with human comprehension. The broader impact of the proposed research includes practical tools for video summarization, scientific advances that appeal broadly to several communities, publicly disseminated research results, inter-disciplinarily trained graduate students, and outreach activities to engage young students in STEM education and career paths.

This project will create a research infrastructure for computer vision and real-time control of autonomous mobile robots (both aerial and ground). The infrastructure includes four integrated components: (1) A Purdue laboratory decorated as miniature cities. (2) Simulators that reflect the physical laboratory. (3) Programmable aerial robots with the same interface as the simulators. (4) Sample solutions for research on artificial intelligence, computer vision, and robot control for evaluation and comparison. This infrastructure will be available to the research community in multiple ways: (1) Users can evaluate their solutions with the simulators in a safe virtual environment. (2) Users can upload their control programs and this team will launch the robots inside Purdue's laboratory. Users can observe the robots remotely using the high-speed cameras already deployed in the laboratory. (3) Users can bring their own robots to the laboratory and conduct experiments. (4) This project will create competitions for researchers to demonstrate their solutions using autonomous mobile robots in simulated emergency and rescue scenarios. The competitions will use miniature buildings and people for the robots to recognize and count objects (such as number of people, vehicles, and houses), assess situations (such as the number of collapsed bridges), while avoiding obstacles.

This infrastructure will be available for investigating a wide range of research topics, including (1) real-time computer vision and control. The decorated laboratory will allow researchers to evaluate their solutions for real-time vision and control methods using active computer vision, navigation, and semantic segmentation in a three-dimensional environment. (2) simulation of robot fleets. Users can evaluate and improve their methods in a safe virtual environment before deployment. (3) This infrastructure will integrate virtual and physical environments so that solutions running in the simulators can be ported directly to the physical robots for experiments. (4) collision avoidance, multi-robot coordination, emergency response, computer security, and efficient machine learning on embedded systems. (5) agriculture, city planning, emergency response, and inspection of civil structures. This project will build STEM talents because autonomous robots and visual data are naturally appealing to the general public. With the simulators, students at all levels can participate without the cost of purchasing physical robots. This research infrastructure will reduce the barriers to innovations. This infrastructure will also encourage innovations in machine learning that are efficient in energy and can be ported to resource constrained embedded systems such as aerial robots. The project will engage a broader audience including K-12 students as well because of the many applications described above.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.