Honglak Lee, Ph.D. - US grants

Machine learning is a powerful tool for artificial intelligence and data mining problems. However, its success critically relies on a good feature representation of the data; therefore, the problem of feature construction poses a fundamental challenge. In recent years, representation learning has emerged as a promising method for learning useful feature representations from data. However, the current state-of-the-art methods are still limited in building intelligent agents that can learn and interact with complex environments and large amounts of sensory input. Specifically, the majority of the existing methods cannot scale well to large-scale data.

The goal of this project is to fill this gap by formulating a new framework that can effectively learn representations from complex environments and scale to large data. Specifically, we propose novel approaches for learning robust representations from large-scale data by (1) controlling the complexity of the feature representations and (2) adaptively modeling relevant patterns in the presence of significant amounts of irrelevant patterns or noise.

Key intellectual contributions of this project will be (1) a novel framework of representation learning that provides robust representations from large amounts of unlabeled data and relatively small amounts of labeled data, and (2) theoretical and algorithmic advances for inference, learning, and related optimization problems in representation learning for large-scale, complex sensory information processing.

This work will serve as a catalyst leading to applications, such as multimedia processing and search, medical image processing, speech recognition, and autonomous navigation. The results will be disseminated through publications and free software.

This research project will create a theoretical and computational infrastructure to design new technology using creativity from crowds and individuals in combination with machine learning. A crowd could be a collection of experts within an organization, a classroom of students, or a large number of people online. Earlier research used machine learning working with individual engineers to help with simple design problems. This research will extend the earlier work to more complex configuration design problems, and will add crowd sourcing. Design representations will be graphs instead of vectors, and the design space will not be defined ahead of time. Machine learning may prove to be an important improvement over design evolution methods, and will provide insight into which design features are important. Machine learning also generates a model of subjective human judgment and preference, leading to more efficient and perhaps more innovative design synthesis.

If successful, this research will provide a platform for new models of innovation with input from multiple stakeholders: A machine supported by crowd-sourced knowledge and real-time interaction with humans will be able to produce unique and creative structures that were beyond the imagination of the humans involved. This research will also result in algorithmic advances in inference, learning, and related optimization techniques in representation learning for structured data and interactive human-machine collaboration. It will also provide a mathematical framework for innovation in massive system design problems involving thousands of designers such as the design of a jet fighter. The technology developed will be possible to implement in a variety of environments from complex engineering system design decisions to ideas for new products on commercial sites. In an educational context, this research, if successful, will allow students and teachers to experiment with design tools, individually, as part of a classroom experience, or as a national endeavor to both learn and possibly to generate new design ideas collectively.

The goal of deep learning is to learn an abstract representation of data with a hierarchical and compositional structure. Deep learning methods can effectively learn discriminative features from high-dimensional input data (e.g., for classification), and have been successfully applied to many real-world problems, such as image classification, speech recognition, and text modeling. Despite these successes, there still remains a challenging open question: how can we learn a robust deep representation that allows for holistic understanding and high-level reasoning from complex data? This CAREER project aims to address this question and is expected to result in novel deep architectures, graphical models, and algorithmic advances for inference, learning, and optimization in deep representation learning. The research outcomes will be disseminated through publications, talks, and tutorials. In addition to advancing the state of the art in deep learning and the many applications it entails, the project will integrate research and education through 1) developing courses in machine learning that include deep learning as a key topic; 2) mentoring significant graduate and undergraduate research activities; and 3) reaching out to K-12 students via hosting demo sessions and mentoring for science fair/research projects.

This project investigates the following closely interrelated and complementary thrusts: First, it develops deep learning algorithms to disentangle factors of variation from complex data. This is done by modeling higher-order interactions between multiple groups of latent variables with a deep generative model (e.g., modeling face images via interaction of latent factors that correspond to identity, viewpoint, and emotion). In addition to better generalization, this approach is amenable to high-level reasoning, such as making analogies. Modeling higher-order interaction will be approached by learning a sub-manifold for each factor of variation, where correspondence information is used for regularizing the latent representation. The project will also develop weakly-supervised and semi-supervised disentangling algorithms that automatically establish correspondences without manual supervision. Second, the project develops deep representation learning methods for structured prediction problems. Specifically, it will develop a graphical model with deep representations that can model complex dependencies between output variables. This framework can be also viewed as data-driven modeling of higher-order prior on structured data, and can be used for modeling higher-order conditional random fields that permit efficient inference and learning. In addition, the project develops stochastic conditional generative models for structured prediction problems that involve uncertainty (i.e., one-to-many mappings). Third, the project develops novel deep learning algorithms for constructing shared representations from multiple heterogeneous input modalities, such as image and text, audio and video, and multiple sensor streams. The main idea is to separately model conditional distribution of each input modality given other modalities. This approach addresses the well-known difficulty of modeling a joint distribution across heterogeneous multimodal input, and provides a theoretical analysis on conditions under which the approach can recover a consistent generative model. This formulation allows for robust recognition and high-level reasoning from heterogeneous multimodal data. Overall, these three thrusts are complementary and are expected to play synergistic roles in tackling a broader range of AI problems and moving beyond the current state-of-the-art in deep learning.

Consider the problem faced by a machine agent that has to interact with some dynamical environment to achieve some goals. Concretely, imagine an agent engaged in a virtual competition as a human would. It can see the screen composed of many moving objects. At any time, it can choose one of a dozen or so actions. Its action controls one of the objects on the screen, but it often is not clear which one. Every so often the an evaluation is given of the competition. At some point the competition ends. How should such an agent choose actions, or more importantly how can we build agents that can learn to compete, i.e., achieve high scores, through trial and error. In this project methods will be developed and evaluated to build such agents.

The above problem is an instance of what is called a reinforcement learning (RL) problem. Such problems abound in sequential decision-making settings. Applications in industry include factory optimization, robotics, and chronic disease management (to list but three diverse domains of interest). Like many of these RL problems, Atari games (used as a testbed here to evaluate learning strategies) have three characteristics of interest to this project. First, they generate high-dimensional images and so the agent faces a difficult perception problem. Second, they often have deeply-delayed rewards; i.e., actions have long-term consequences. For example, losing a resource may not cost at the moment of loss, but could lead to very high losses much later when that resource is critically necessary. Third, they have deep partial observability, i.e., to compete effectively one has to often remember the deep past. For example, a location encountered far back in the past may become valuable much later because a critical resource becomes available at that time and the agent would have to find its way back to that location to use the resource. It is proposed to address these three challenges respectively with new neural network architectures for predicting the consequences of actions, new methods for intrinsically motivating agents even when reward is delayed, and new recurrent neural network architectures to remember the past effectively. Success of the proposed work is expected to significantly expand the scope of application of reinforcement learning. Finally, Atari games will be used instead of, say, factory optimization as an evaluation domain because they are readily available. They will be used to draw high-school and under-represented undergraduate students interest into complex ideas underlying the proposed work; their fun visualizations will allow them to be integrated into teaching in the PIs' classes, and there are a variety of games that vary in the degree of difficulty of the three challenge dimensions allowing more effective control of the evaluations more effectively.

The construction industry could benefit significantly from increased productivity, use of modern work processes, improved relationships among stakeholders, and increased attention to safety and health issues of construction workers. In the absence of these improvements, a chronic shortage of skilled construction workers has developed, largely due to a retiring workforce and the reluctance of younger generations or people of different abilities to pursue such careers. More recently, the outbreak of the COVID-19 pandemic has caused serious economic difficulties and schedule delays on construction projects, since it is hard to maintain social-distancing on construction sites. These impacts further emphasize the need for construction techniques that can allow workers to perform tasks remotely, decreasing the number of on-site workers at any one time to ensure worker health and safety. This planning grant will support investigation of the potential of human-robot teams to transform future construction work and the profile of future construction workers. It is expected that such changes would result in new career opportunities and significant benefits to the industry. The project team envisions that human workers will use technology to teach co-robots to perform construction work tasks remotely, resulting in symbiotic human-robot teams that can be widely deployed in the construction industry. This approach parallels the classical Master-Apprentice vocational model prevalent in today?s construction industry.

The overarching goal of this research planning grant is to explore the feasibility and potential of a human-robot team through engagement with a range of stakeholders. The proposed activities for this planning project include: 1) Fact-finding surveys distributed to representatives of construction firms, current construction workers, and potential future construction workers (high school students); 2) Technology pilot presentation and feedback from representatives of construction firms and workers; and 3) A comprehensive research program development workshop with expert stakeholder participants from academia and industry. These activities will allow the research team to set the foundation for developing a convergent research agenda that reshapes the future of construction work into a human-robot partnership that supports a self-sustaining cycle of lifelong learning, knowledge-transfer, and effective teamwork. Ultimately, the results of this project will inform the opportunities and challenges of incorporating co-robots on construction sites without replacing the current workforce or adversely impacting the construction work process. The success of this planning grant will pave the way for a significant research effort that takes a human-centered approach toward the future of construction work and ensures that it improves accessibility, quality of life, productivity, and job satisfaction. The PIs will recruit a diverse group of students and partners from industry and academia and pursue broad dissemination of the research findings in academic and industry venues. This project has been funded by the Future of Work at the Human-Technology Frontier cross-directorate program to advance design of intelligent work technologies that operate in harmony with human workers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.