Yiming Ying - US grants

Although existing machine learning (ML) methods are effective for analyzing data that are static in nature, many of today's applications from information retrieval to web searching require real-time processing of massive amounts of streaming data. Often, the data is available in an online fashion, meaning the entire dataset is not available at once, but rather, individual data instances arrive sequentially. Area under the Receiver Operating Characteristic Curve (AUC) has been proven to be an effective evaluation metric and learning objective in many application domains, but it is difficult to optimize directly. To date, there are no satisfactory approaches to incorporate AUC into online ML algorithms for classification and ranking problems. This project will address the fundamental theoretical and algorithmic challenges in ML algorithms based on AUC maximization for processing streaming, high-dimensional data with efficient algorithms and theoretical analysis. The results of this project are expected to have broader impact in intrusion detection for cyber-security, fault detection in safety critical systems, information retrieval and cancer diagnosis. The planned research will also integrate with educational activities, including developing new undergraduate/graduate courses on optimization and machine learning, organizing a workshop and making software tools freely available to the public. The principal investigators also plan to undertake outreach activities to improve STEM learning of high school students.

A central topic of this project is to develop efficient online learning algorithms for AUC maximization and bipartite ranking, making them amenable for online processing of high dimensional and large volume of streaming data. This project also establishes a rigorous statistical foundation and thorough theoretical analysis of the online AUC maximization algorithms developed. The primary technical challenge in developing online AUC maximization algorithms and theory is that the objective functions involve statistically dependent pairs of instances while individual instances continuously arrive in a sequential order. This is in contrast to standard classification based on accuracy, where the loss functions only depend on individual instances and the related algorithms and theory are well developed. The project will address this gap by sufficiently exploiting the structures of the population objective involved in the problem of AUC maximization, rather than an empirical objective on finite data, through novel interaction of machine learning, statistical theory and applied mathematics. This will help to remove the pairwise structure in the original objective function and facilitate the development of efficient online learning algorithms for AUC maximization.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Machine learning is instrumental for the recent advances in AI and big data analysis. They have been used in almost every area of computer science and many fields of natural sciences, engineering, and social sciences. The central task of machine learning is to ?train? a model, which entails seeking models that minimize certain performance metrics over a set of training examples. Such performance metrics are termed as the aggregate losses, which are to be distinguished from the individual losses that measures the quality of the model on a single training example. As the link between the training data and the model to be learned, the aggregate loss is a fundamental component in machine learning algorithms, and its theoretical and practical significance warrants a comprehensive and systematic study. The proposed work will focus on several fundamental research questions concerning the aggregate loss: are there any other types of aggregate loss beyond the average individual losses?; if so, what will be a general abstract formulation of these new aggregate loss?; how can the new aggregate losses be adapted to different machine learning problems?; and what are the statistical and computational behaviors of machine learning algorithms using the general aggregate losses?.

The technical aims of the project are divided into four interrelated thrusts. The first thrust explores new types of rank-based aggregate losses for binary classification and study efficient algorithms optimizing learning objectives formed based upon them. The new aggregate losses will be applied to problems such as object detection, where rank-based evaluation metric is used dominantly. The second thrust aims to deepen our understanding of the binary classification algorithms developed using the rank-based aggregate losses and will be focused on a study of their statistical theories such as generalization and consistency. The third thrust will extend the study of new types of aggregate losses to other supervised problems (multi-class and multi-label learning and supervised metric learning) and unsupervised learning. The fourth thrust dedicates to the theoretical aspects of aggregate losses, in which an aggregate loss will be abstracted as a set function that maps the ensemble of individual losses to a number. This abstraction will be exploited to study the properties of new aggregate losses that make them superior than the average loss and propose new aggregate losses beyond rank-based ones.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project promotes the progress of science and technology development by advancing artificial intelligence (AI) through innovations in scalable and robust computational methods. AI, especially deep learning, has brought transformative impact in industries and quantum leaps in the quality of a wide range of everyday technologies including face recognition, speech recognition and machine translation. However, in order to accelerate the democratization of AI there are still many challenges to be addressed including data issues and model issues. This project seeks to advance AI by addressing one critical issue related to data; i.e., data imbalance. This happens when the collected data for training AI models does not have enough instances representing some property the models are trying to learn. For example, molecules with a certain antibacterial property would be far fewer than all possible molecules making predictions of antibacterial properties challenging. The goal of this project is to develop algorithms with theoretical guarantees to make AI learn more effectively from the big imbalanced data. This project will also contribute to training future professionals in AI and machine learning, including training high school students and under-represented undergraduates.

This project investigates a broad family of robust losses for deep learning. The research activities include (i) developing scalable offline stochastic algorithms for solving non-decomposable robust losses that are formulated into min-max, min-min formulations; (ii) developing efficient online stochastic algorithms for solving a family of distributionally robust optimization problems that are cast into compositional optimization problems; (iii) developing effective strategies for training deep neural networks by solving the considered non-decomposable robust losses; (iv) establishing the underlying theory including optimization and statistical convergence of the proposed algorithms. The algorithms are being evaluated on big imbalanced data such as images, graphs, texts.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The paradigm of learning from data is playing an increasingly important role in science and engineering. The interplay between machine learning and mathematical optimization has been most fruitful, and one prominent area is stochastic convex optimization (SCO). However, there is relatively little work on the fundamental questions such as generalization and stability analysis, and the existing studies often focus on the standard classification and regression with smooth losses. Furthermore, data collected and used for the learning often contains sensitive information such as financial records from fraud detection or genomic data from cancer diagnosis which presents an urgent need to develop privacy-preserving SCO algorithms with theoretical guarantees. These provide motivation for the project which aims to study the fundamental properties of machine learning inspired SCO algorithms including their stability, generalization, and differential privacy. Students will be involved and trained in interdisciplinary aspects.

The technical objectives of the proposed work are divided into three thrusts. The first thrust focuses on the study of stability and generalization of stochastic gradient methods (SGM) for solving SCO problems associated with non-smooth losses. The second thrust is to develop and study SGM algorithms for SCO problems which can prevent the privacy leakage using a well-accepted mathematical definition of privacy called differential privacy. The third thrust is to study the stability, generalization, and differential privacy of SCO algorithms for pairwise learning which involves more complex losses than the standard classification and regression.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.