Yezhou Yang - US grants

The hand is one of the most complex and beautiful pieces of natural engineering in the human body, and it represents a triumph of complex engineering, exquisitely evolved to perform a range of tasks. Hand-gestures have been used for many areas and how to model hand-gestures has broad and profound impact on addressing the nation's priorities and societal needs, e.g., manipulators and co-robot in manufacturing, natural human-computer interaction for virtual reality, wearable platforms in consumer electronics, hand-gesture biometrics in cybersecurity, etc. In this project, one of the research goals is to build a hand-gesture research platform, Hand-Gesture Research Platform (HGRP). HGRP targets at enabling researchers to easily access various hand-gesture data to validate their hand-gesture recognition models, benchmark the performance of newly developed algorithms, and compare with research outcomes from others. Moreover, HGRP is used to gather research communities' feedback based on existing cutting-edge hand-gesture research to prioritize the need on establishing a hand-gesture focused computing research infrastructure.

The HGRP framework is based on a cloud computing platform and is used to enable research capabilities in the following areas: (a) hand-gesture biometrics; (b) cognitive Robotics; and (c) programmable interfaces for gesture-based data processing and visualization. HGRP is composed by the following salient features:

*Data collection based on two major types of sensors: (a) motion detection sensors, e.g., wearable sensors such as watch, wrist band, on figure sensors, data motion gloves, infrared motion detection sensors, etc., and (b) video sensors such as leap motion sensors, video recorders, etc. The detected hand-gesture data is sent to data storage for processing and storing.

*Data are collected and stored on an objective storage service, and frequently used data is stored in memory storage.

*A GPU-based private cloud is established to allow researchers to implement well- known hand-gesture data processing models and establish benchmarking models.

HGRP also allows researchers to submit computation tasks for evaluations and comparative studies. HGRP provides a web-based data collection, processing, sharing, and storing APIs that allow researchers remotely to access the hand-gesture repository for data retrieval, processing, sharing, storing through web services APIs.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The goals of Automated Driving Systems (ADS) and Advanced Driver Assistance Systems (ADAS) include reduction in accidental deaths, enhanced mobility for differently abled people, and an overall improvement in the quality of life for the general public. Such systems typically operate in open and highly uncertain environments for which robust perception systems are essential. However, despite the tremendous theoretical and experimental progress in computer vision, machine learning, and sensor fusion, the form and conditions under which guarantees should be provided for perception components is still unclear. The state-of-the-art is to perform scenario-based evaluation of data against ground truth values, but this has only limited impact. The lack of formal metrics to analyze the quality of perception systems has already led to several catastrophic incidents and a plateau in ADS/ADAS development. This project develops formal languages for specifying and evaluating the quality and robustness of perception sub-systems within ADS and ADAS applications. To enable broader dissemination of this technology, the project develops graduate and undergraduate curricula to train engineers in the use of such methods, and new educational modules to explain the challenges in developing safe and robust ADS for outreach and public engagement activities. To broaden participation in computing, the investigators target the inclusion of undergraduate women in research and development phases through summer internships.

The formal language developed in this project is based on a new spatio-temporal logic pioneered by the investigators. This logic allows one to simultaneously perform temporal reasoning about streaming perception data, and spatially reason about objects both within a single frame of the data and across frames. The project also develops quantitative semantics for this logic, which provides the user with quantifiable quality metrics for perception sub-systems. These semantics enable comparisons between different perception systems and architectures. Crucially, the formal language facilitates the process of abstracting away implementation details, which in turn allows system designers and regulators to specify assumptions and guarantees for system performance at a higher-level of abstraction. An interesting benefit of this formal language is that it enables querying of databases with perception data for specific driving scenarios without the need for the highly manual process of creating ground truth annotations. Such a formal language currently does not exist, and this is a huge impediment to building a thriving marketplace for perception components used in safety-critical systems. This framework sets the foundation for a requirements language between suppliers of perception components and automotive companies. The open source and publicly available software tools developed in this project will assist with testing of perception systems by engineers and governmental agencies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Generative models describe real-world data distributions such as images, texts, and human motions, and are playing an essential role in a large and growing range of applications from photo editing to natural language processing to autonomous driving. There are two open challenges regarding the development and dissemination of generative models: (1) Adversarial applications of generative models have created concerning socio-technical disturbances (e.g., espionage operations and malicious impersonation); and (2) developing generative models using multiple proprietary datasets (which are needed to reduce data biases) raises privacy concerns about data leakage. Legislative efforts have recently been taken in the wake of these challenges, so far with limited consensus on the format of regulations and knowledge about their technological or social feasibility. To this end, this project will develop new mathematical theories and computational tools to assess the feasibility of two connected solutions to these challenges: Model attribution enforces the owners to be correctly identified based on their generated contents; secure training ensures zero data leakage during the collaborative training of attributable generative models. If successful, the outcomes of the project will provide technical guidance for future regulation design towards secure development and dissemination of generative models. Project results will be disseminated through a project website, open-source software, and public datasets. The impacts of the project will be broadened through educational activities, including new course modules on Artificial Intelligence (AI) security, undergraduate research projects, and outreach to the local community through lab tours, to prepare underrepresented groups with skills to mitigate risks from malicious impersonation and biased data/model representations targeting these groups.

This project will focus on synergistic research tasks towards decentralized model attribution and secure training of generative models. In the former, the research team will study the systematic design of a set of user-end generative models that can be certifiably attributed by a set of binary classifiers, which are stored in a decentralized manner to mitigate security risks. The technical feasibility of decentralized attribution will be measured by the tradeoffs between attributability, generation quality, and model capacity. In the latter, the research team will study secure multi-party training of generative models and the associated binary classifiers for attribution. Data privacy and training scalability will be balanced through the design of security-friendly model architectures and learning losses. New knowledge will be created that differentiates this project from the existing state-of-the-art literature in digital forensics and secure computation: (1) Sufficient conditions for decentralized attribution will be developed, which will reveal analytical connections between attributability, data geometry, model architecture, and generation quality. (2) The sufficient conditions will enable estimation of the capacity of attributable models for a given dataset and generation quality tolerance. (3) Feasibility of sublinear secure vector multiplication will be studied, which will fundamentally improve the scalability of secure collaborative training. (4) Privacy-friendly activation and loss functions will be designed for the training of user-end generative models and the classifiers for attribution.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Intelligent systems that can robustly process vision and language data are necessary to enable integrated AI applications (such as automated driving, robotic home assistant, etc.) and improve quality of life. However, such systems typically operate in open and highly uncertain environments for which physical and geometric understanding, semantic robustness, and conducting hypothetical reasoning become essential. This project will result in a publicly available software suite that can assist with training and validating robust Vision and Language (V&L) systems. In particular, the resulting semantic transformations will be packaged as an API service that companies and universities could quickly utilize. The resulting benchmark challenges will be made publicly available for further V&L research. Finally, the proposed study will stimulate educational activities at ASU in training graduate and undergraduate students in AI/ML/CV/NLP with a "post-dataset era'" vision. The project will also train 2 Ph.D. students and several master-with-thesis students, develop a new seminar course, recruit underrepresented minority participants at all levels, and reach K-12 students with modules that explain the challenges in developing robust intelligent systems.

Robust intelligent systems such as home assistant robots fundamentally depend on highly correlated vision and language systems and fine-grained data alignment. Even though the existing approaches demonstrate success on carefully collected benchmarks, it is not sufficient to establish robustness, reliability, and out-of-distribution generalization for them to be deployed in real-world applications. The project will conduct a systematic study on intelligent and active data engineering to boost their performance and robustness. By investigating a novel and active perspective towards vision and language data engineering, the project will address the following three fundamental research tasks: 1) development of data generators to hallucinate training data from existing ones with low-level vision; 2) with hypothetical actions, and 3) design of training paradigms incorporating the new data generated with the goal of increasing the ultimate systems' generalization capability and robustness.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.