Trevor Darrell, Ph.D. - US grants

This is funding to support attendance by approximately 15 graduate students in a doctoral consortium (workshop) to be held in conjunction with the Sixth International Conference on Multimodal Interfaces (ICMI'04), to be held October 13-15, 2004 in State College, PA, and sponsored by the Association for Computing Machinery (ACM). The 3-day conference will bring together researchers from academia and industry from around the world to present and discuss the latest multi-disciplinary work on multimodal interfaces, systems, and applications. The conference represents the growing interest in next-generation perceptive, adaptive and multimodal user interfaces. These new interfaces are especially well suited for interpreting natural communication and activity patterns in real-world environments; their emergence represents a radical departure from previous computing, and is rapidly transforming the nature of human-computer interaction by creating more natural, expressively powerful, flexible and robust means of interacting with computers. Participants in the doctoral consortium will receive feedback from an invited committee of 3-5 senior personnel to posters which reflect work-in-progress not mature enough for a full paper at ICMI.

Broader Impacts: The doctoral consortium will give students exposure to their new research community, both by presenting their own work and by observing and interacting with established professionals in the field. It will encourage students at this critical time in their careers to begin building a social support network of peers and mentors. The organizers will make a special effort to recruit student participants from less well-funded institutions, where adequate support for student conference attendance may be otherwise unavailable. Participation by members of traditionally under-represented groups, including women, minority students, people with disabilities and veterans, will be strongly encouraged.

This is funding to support attendance by approximately 15 graduate students in a doctoral consortium (workshop) to be held in conjunction with the Eighth International Conference on Multimodal Interfaces (ICMI'06), to be held November 2-4, 2006 in Banff, Canada, and sponsored by the Association for Computing Machinery (ACM). ICMI is the foremost conference representing the growing interest in next-generation perceptive, adaptive and multimodal user interfaces, systems, and applications, which are especially well-suited for interpreting natural communication and activity patterns in real-world environments; their emergence represents a radical departure from previous computing, and is rapidly transforming the nature of human-computer interaction by creating more natural, expressively powerful, flexible and robust means of interacting with computers. The 3-day conference will bring together researchers from academia and industry from around the world to present and discuss the latest multi-disciplinary work in the field. The theme of this year's conference is multimodal collaboration through different platforms and applications; the conference will focus on major trends and challenges in this area, including distilling a roadmap for future research and commercial success. Participants in the doctoral consortium will receive feedback from an invited committee of half a dozen senior personnel to posters which reflect work-in-progress not mature enough for a full paper at ICMI; they will also get to present their work orally to the conference as a whole during a doctoral spotlight session highlighting top student research

Broader Impacts: The doctoral consortium will give students exposure to their new research community, both by presenting their own work and by observing and interacting with established professionals in the field. It will encourage students at this critical time in their careers to begin building a social support network of peers and mentors. Participants will be selected with the goal of increasing the conference's breadth and diversity; the organizers will make an effort to recruit members of traditionally under-represented groups including women, minorities, and people with disabilities, as well as from less well-funded institutions where adequate support for student conference attendance may be otherwise unavailable.

Humans naturally use dialog and gestures to discuss complex phenomena and plans, especially when they refer to physical aspects of the environment while they communicate with each other. Existing robot vision systems can sense people and the environment, but are limited in their ability to detect the detailed conversational cues people often rely upon (such as head pose, eye gaze, and body gestures), and to exploit those cues in multimodal conversational dialog. Recent advances in computer vision have made it possible to track such detailed cues. Robots can use passive measures to sense the presence of people, estimate their focus of attention and body pose, and to recognize human gestures and identify physical references. But they have had limited means of integrating such information into models of natural language; heretofore, they have used dialog models for specific domains and/or were limited to one-on-one interaction. Separately, recent advances in natural language processing have led to dialog models that can track relatively free-form conversation among multiple participants, and extract meaningful semantics about people's intentions and actions. These multi-party dialog models have been used in meeting environments and other domains. In this project, the PI and his team will fuse these two lines of research to achieve a perceptually situated, natural conversation model that robots can use to interact multimodally with people. They will develop a reasonably generic dialog model that allows a situated agent to track the dialog around it, know when it is being addressed, and take direction from a human operator regarding where it should find or place various objects, what it should look for in the environment, and which individuals it should attend to, follow, or obey. Project outcomes will extend existing dialog management techniques to a more general theory of interaction management, and will also extend current state-of-the-art vision research to be able to recognize the subtleties of nonverbal conversational cues, as well as methods for integrating those cues with ongoing dialog interpretation and interaction with the world.

Broader Impacts: There are clearly many positive societal impacts that will derive from this research. Ultimately, development of effective human-robot interfaces will allow greater deployment of robots to perform dangerous tasks that humans would otherwise have to perform, and will also enable greater use of robots for service tasks in domestic environments. As part of the project, the PI will conduct outreach efforts to engage secondary-school students in the hope that exposure to HRI research may increase their interest in science and engineering studies.

Learning visual representations remains a challenge for systems which interact with the real world and/or analyze visual information available on the web. Significant progress has been made with simple visual features based on gradient histograms: these models work extremely well on objects that have highly textured and nearly planar patterns or parts. However, these systems suffer when faced with certain classes of real world objects that do not have discriminative locally-planar opaque texture patches, especially objects with complex photometric models. This project develops layered visual models for visual recognition, which can model these classes of phenomena. The research team grounds the methods in a probabilistic foundation, primarily exploiting a sparse Bayesian approach to factoring observed image features into a set of component layers corresponding to an additive image formation process. Considering both local descriptor and local feature detector variants of the model, the research team offers a new concept for interest point detection in the case of transparent objects: extrema detection in a latent-factor scale space. This model has the potential to find invariant local detections despite transparency, and could be useful in a range of vision applications beyond pure recognition for which sparse local feature detectors have proven valuable (e.g., registration, mosaicing, SLAM). Robotic vision systems can use this representation for enhanced recognition of everyday objects, supporting domestic and industrial applications. These representations also facilitate intelligent media processing and indexing.

This project improves autonomous robotic perception so that future co-robots can glance around any scene and accurately estimate how it would feel to grasp or step on all of the visible surfaces. Just as people do, robots should use such these physical predictions to guide their interactions with the world, for example avoiding dangerous ice patches on the ground when walking and driving, and adeptly anticipating the grasp force needed to pick up everything from ice cubes to stuffed animals. These research activities are accompanied by significant outreach efforts, including a new program on "Look and Touch Robotics" to get middle-school students, particularly those from under­represented groups, excited about computer science, engineering, and robotics. This program uses simple experiments to highlight the dual importance of visual and haptic information during interactions with physical objects, along with demonstrations of a robot showing visuo-­haptic intelligence. This project also integrates research and education by involving undergraduates in the research and via hands-on projects in the vision and robotics classes taught by the Principal Investigators.

This research involves extensive collection of data from real objects and surfaces using both visual and haptic sensors. The recorded interactions are analyzed to uncover visual clues that can allow a robot to infer the physical characteristics of the surface, such as slipperiness, hardness, and roughness. This problem is addressed using deep learning, a recently developed approach that has been successful in enabling robots to visually recognize a wide variety of objects in diverse circumstances. The research team also builds the database of visuo-haptic recordings and the learned cross-modal sensory, and makes it available to other robotics researchers at the end of the project.