Thanassis Rikakis - US grants

Over the past decade, human motion analysis has become an important research area with critical applications. It is attracting significant research efforts in a number of disciplines, such as computer vision (vision-based motion capture, human computer interface, human identification), robotics (navigation), dance and choreography (automatic dance documentation and dance instruction), music (digital conducting) and bioengineering (rehabilitation and motor behavior). Motion analysis is a complex problem due to the 3D nature of the human body; the infinite possibilities of human movements; variability of movement execution between different people; continuously adaptive learning through feedback from and interactions with the environment; and the inherent multiple levels of movement structure in terms of time, space and energy. This makes it unrealistic for a single discipline to address all aspects. Therefore, progress within each discipline moves at a slow pace.
Intellectual Merit: Arizona State University has founded the Interdisciplinary Research Environment for Motion Analysis (IREMA) initiative that integrates researchers from ten disciplines to create a holistic model for motion analysis research and education. Within IREMA, ground-breaking collaborations have been established through networks of experts, infrastructures and important applications. Using this multi-level, networked research model, the principal investigators (PIs) are able to address many critical issues of real-time motion capture, analysis and feedback. Promising results of social significance are being achieved in areas such as: Rehabilitation Research to Restore Functional Walking Ability for Spinal Cord Injured, Auditory Display Systems for Aiding Interjoint Coordination, Modeling of Human and Robotic Heuristics for Projectile Interception, Movement Based Interactive Arts Environments, Experiential learning environments for children, Extraction and Recognition of Middle and Low Level Features of Movements, Vision-based Motion Capture Using Domain Knowledge.
Using the research infrastructure (RI) grant the PIs will create a multimodal sensing and feedback environment for human motion analysis research and movement-based interactive applications. They will increase their optical motion capture system to 24 cameras, create a high-speed, high resolution 24 video camera array, complete the building of a pressure sensitive floor, acquire a new EMG system and metabolic sensing equipment, acquire required hardware to integrate optical motion capture data with EMG and 2D visual as well as metabolic sensing, increasing processing and storage capacity, creating a mobile motion capture setup, and deploying the necessary hardware and software for interactive real-time feedback. The above sensing equipment would provide high-speed, high quality, synchronous video capture of multiple views, high-precision marker-based motion capture and pressure sensing in the floor as well as on the treadmill, and audio signals. It will enable the PIs to capture human movement in its full essence. The optical motion-capture data and the pressure sensing data will be fused to provide holistic motion capture. The processed, combined data of these systems will be used to train the video based system so that robust and accurate vision-based motion-capture can be acquired using low-cost video cameras. The physiological equipment will be used in the rehabilitation projects.
Broader Impact: During this five-year project, the PIs hope to achieve major advances in motion analysis and core computer science areas: computer vision, human-computer interaction, information and data management, geometric computation, knowledge systems and robotics. These advances will have significant social impact by producing major progress in movement rehabilitation and therapy, K-12 education, security applications (gait/face recognition), and all areas involving movement training (dance, theatre, sports, firefighting, military). Finally, IREMA can serve as a new model for research and interdisciplinary collaboration, which can be adapted to other areas thereby increasing their productivity. This RI grant will establish the necessary infrastructure for paradigm shifts in motion analysis and will facilitate the overall modeling of hybrid research.

This IGERT award at the Arts, Media and Engineering Program at Arizona State University will develop research and training mechanisms for the creation of a new class of media scientists. These scientists will produce new approaches for the integration of computational elements and digital media in the physical human experience. Their work will result in experiential media systems - hybrid physical-digital environments that address significant challenges in key areas of the human condition such as health, education and everyday living.

The knowledge required to create experiential media systems is currently fragmented across engineering, sciences and arts. This IGERT award will train a new generation of hybrid media engineers-scientists-artists who are equipped to transcend this fragmentation. The training will be realized through a large interdisciplinary network combining expertise from twelve contributing disciplines. This network will allow integrated advanced research in sensing, modeling, feedback, experiential construction and learning. The research will result in new knowledge in media systems as well as within each contributing area. It will also result in the development of large-scale applications of societal significance. The graduate training mechanisms are implemented through formally approved concentrations within the graduate degree programs of participating disciplines. They combine discipline specific education in one of the IGERT research areas with interdisciplinary training in media development. The framework of this IGERT allows for methodology found in the sciences to be combined with creativity found in the arts. It will bridge the gap between computation and the physical experience, advance human-centric technologies and produce major advances in education, rehabilitation, communication, and everyday living. IGERT is an NSF-wide program intended to meet the challenges of educating U.S. Ph.D. scientists and engineers with the interdisciplinary background, deep knowledge in a chosen discipline, and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing innovative new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries.

The goal of this workshop is to collaboratively determine successful strategies for promoting creativity in IT based on the experiences of a diverse group of leading practitioners. Creative activity leading to major innovation is a critical component of the American competitiveness initiative, not only for driving innovation in IT research and development, but also for advancing our pedagogical approaches to educating and preparing scientists for the future. There is a need to establish a formal collective basis for critically examining the role of creativity in IT research and education in order to support structured dialog, disseminate findings and broaden understanding.

To begin this process, this workshop brings together leaders engaged in fostering creativity within a variety of disciplines to share their experiences and success stories. Topics addressed will include diverse perspectives on fostering creativity; integrating creative and scientific disciplines; formalizing and measuring success factors in encouraging creativity; and creativity through the exchange of ideas in electronic publication forums. The presentations and the surrounding conversations that will shape, extend and evolve common ideas will be collected into an edited electronic publication that will be part of the dynamic creativeIT wiki.

Collaborative Projects: EAGER: A virtual eXchange to support networks of creativity and innovation amongst Science, Engineering, Arts and Design (XSEAD)

Intellectual Merit
One of the greatest challenges facing the United States in research and education is how to fundamentally encourage innovation across all sectors and spawn new solutions to address global challenges. Increasing research evidence and industrial innovations (i.e. mobile computing, social media) confirm that broad interdisciplinary collaborations that include both science and art fields have great potential for spawning creativity and innovation in computer science, engineering and the sciences. An emerging hybrid community of scientists, engineers, artists and designers is producing innovative and entrepreneurial research that advances new knowledge and proposes holistic solutions to societal challenges including health, education and environmental change. Yet, this burgeoning interdisciplinary community continues to face problems in its efforts to self-organize among constraints imposed by academic systems and historical biases; it continues to seek a dynamic and synergizing research and outreach exchange.

Building upon lessons-learned, a new Virtual eXchange to support networks of creativity and innovation amongst Science, Engineering, Art and Design (XSEAD) will be developed. The XSEAD project will address the following urgent needs of the interdisciplinary science-art community: establish a cohesive view of the field and provide a mechanism to attract entrepreneurs and industry; create a venue for multimodal documentation of research outcomes; provide extensive databases of prior and current research; allow rapid dissemination of research outcomes; facilitate forming of collaborations and specialized sub-communities; document and help evolve science-art curricula efforts and evaluation approaches; provide context and support mechanisms for science-arts careers; establish evidence of the societal impact of interdisciplinary science-art integration. The software engineering development components of XSEAD will contribute further knowledge in three technical areas: Content organization (improve the effectiveness of algorithms for dynamic, usage based, organization of large multimedia databases); Recommendation algorithms (promote the use of multi-relational structures for providing effective recommendations); Community dynamics (develop novel algorithms to extract structures that encode meaningful interactions in online social networks).

Broader Impact
XSEAD will expose general non-expert audiences to the evolution and potential of collaborative research across science and arts. It will attract the interest of young people searching for careers that combine the rigor of science and engineering with the creativity and reflection of arts and design. It will serve teachers and informal learning communities seeking exemplars for curricular development, active practitioners looking for further institutional opportunities to present and support their ongoing work, academics developing related interdisciplinary efforts and commercial companies seeking cross-trained expertise. XSEAD will enable rapid research exchange and in-depth peer-reviewed scholarship between the worlds of science and art and provide a unique and deeply engaging inroad to a vast and creative repository. XSEAD will help promote new paradigms for developing human centric solutions to complex societal problems (i.e. cost effective health and wellness, globalization and conflict, adaptive K-12 learning, electronic communication and security). These paradigms will combine knowledge across broad and diverse areas of human knowledge.

With the aging of the US population, there is an increasing need for effective and accessible rehabilitation services for debilitating illnesses and injuries such as stroke and arthritis. Intensive long-term rehabilitation is challenging to administer in an accessible and affordable way as it requires frequent trips to the clinic (usually supported by a caregiver), and significant one-on-one time with rehabilitation experts. Telemedicine and telehealth are gaining prominence as cost effective ways to deliver home-based health and wellness to wider populations. However, automated tele-rehabilitation is not currently feasible as the expert functions of the therapist cannot yet be fully automated and replicated in the home. In addition, there are significant technical, behavioral, and clinical challenges to scaling technology assisted home-based rehabilitation. This project aims to address these challenges through the development of a system for Semi-Automated Rehabilitation At Home (SARAH). The system is defined as semi-automated because it relies on the remote participation of the therapist for developing and adapting the therapy program. The SARAH system uses the remote therapists? instructions to guide the patient through daily intensive therapy sessions at the home. Using inexpensive sensing technologies that are non-intrusive and mindful of the patient?s privacy, the system records and analyzes the daily therapy sessions as well as the general activities of the patient in the home. The SARAH system then provides feedback to the patient based on their therapy activities and general movements around the home. The system also provides summaries of patient progress to the remote therapist so that they can adapt the program for subsequent therapy sessions. The first version of the SARAH system focuses on upper extremity stroke rehabilitation at the home as the team of researchers has significant experience in this space. Additional outputs from this project, including the development of a generalized system and relevant methodology, are designed to support a wide variety of home-based rehabilitation contexts.

The technical goals of the project are the development of movement assessment algorithms fusing knowledge based and data driven approaches. This fused approach produces automated patient assessment feedback during home-based therapy, and summaries of patient therapy and daily activities to assist the therapist with remote decision making. The project utilizes a Hierarchical Bayesian Model (HBM) approximating the therapist decision process as a common framework for the development of integrative cyber-human movement assessment algorithms. Therapy sessions are captured using two video cameras and four wearable Inertial Measurement Units (IMUs), while daily activity is only be tracked through the IMUs to estimate the wearer's 3D kinematics. The project fuses clinician?s expert knowledge of therapy tasks and segments with video and IMU data to implement automated segmentation and rating of therapy at the home. The fused cyber-human assessment of therapy data is used to inform the translation of low-level IMU feature tracking during daily life activities into daily movement summaries assisting remote therapy assessment and customization. The automated summaries include: therapy adherence, quality of therapy performance, quantity of patient daily activity and movement in the house, use of impaired limb, tasks detected during daily activity, and confidence of identification. The fusion of knowledge based and data driven approaches for computational movement analysis, as well as the cyber-human design process itself, will yield higher-level generalizable insights extending to many more applications of machine learning and deep learning in data-constrained scenarios. The low-cost sensor networks and wearable sensor solutions produced by the project will provide practical ways to monitor kinematics in real-world environments such as improved control systems for prosthetics and exoskeletons, prevention of workplace injuries through biofeedback, and enhancements in human-robot collaboration.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.