Matthew Turk - US grants

This project focuses on facilitating and augmenting social interaction in virtual environments, particularly immersive virtual environments. Virtual environment technology allows individuals to freely move about digital "worlds" in real time observing and interacting with the environment and
virtual others within it. Increased sophistication of virtual environment technology and digital imaging of people promises a new age for technologically mediated social interaction of geographically separated individuals. However, in order to implement such interaction virtually in
meaningful and productive ways, an understanding of the parameters of people's perceptions of each other's non-verbal signals (e.g., facial expressions, gestures, gaze) within virtual environments is necessary. Such an understanding will provide a hierarchical taxonomy of the necessary and
sufficient non-verbal signals that are critical to social interaction within virtual environments and, therefore, must be tracked and rendered among interactions in virtual environments. Realizing the objectives of the proposed project will advance scientific understanding in the areas of social interaction and non-verbal behavior, human participation in collaborative virtual environments, and technological (e.g., computer vision) aspects of automated tracking and rendering of human on-verbal signals.

This IGERT award will support the establishment of a multidisciplinary graduate training program of education and research at UCSB in the broad field of interactive digital multimedia. The objective of the program is to train students in a truly interdisciplinary approach to digital multimedia, while preparing them for careers in industry, research, and education. The convergence of digital media, computing, and communication has created new and exciting opportunities in science, engineering, and the arts. The goal of this program is to integrate multiple, diverse approaches to the creation, analysis, deployment, and utilization of digital media within a coherent educational framework of training and research. This multidisciplinary endeavor necessitates collaboration between students and faculty from a broad spectrum of backgrounds and perspectives, including computer science, electrical and computer engineering, psychology, geography, design, composition, and art. Through innovative and interdepartmental courses and seminars, opportunities for internships in industry, and guidance by faculty from several departments, students will gain a unique perspective on interactive multimedia, including the creation, encoding, and distribution of multimedia content, as well as applications of multimedia systems in education, communication, and arts and entertainment. Areas of focus for research and education include image, video, and audio processing, networking, human-computer interaction, graphics and visualization, visual arts, interactive media, and computer music. The graduate training program will emphasize a broad background in several of these areas, while research projects, interactive installations and artistic performances will bring together small, interdisciplinary teams of researchers with diverse expertise. The IGERT-supported program will leverage existing strengths at UCSB in related areas and will involve the participation of several departments. The program will complement the UCSB Media Arts and Technology Program, a masters-level graduate program, by enabling doctoral students from several departments to build upon the current foundation it provides.

IGERT is an NSF-wide program intended to meet the challenges of educating U.S. Ph.D. scientists and engineers with the multidisciplinary backgrounds and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing innovative new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries. In the fifth year of the program, awards are being made to twenty-one institutions for programs that collectively span the areas of science and engineering supported by NSF.

The research in this project will develop robust and effective algorithms for detecting and analyzing meaningful image discontinuities, which are critical cues that play an important part in building general image understanding systems. The project will build on promising preliminary research on multi-flash imaging, which uses active illumination to achieve robust detection and labeling of depth discontinuities in images. These ideas will be generalized by varying various aspects of the illumination parameters and developing methods to handle a much wider variety of imaging conditions. The research will develop a framework for robust multi-view stereo by integrating viewpoint variation with active illumination, and it will extend the concept to develop methods for detecting and analyzing other kinds of meaningful discontinuities (in surface normal, illumination, motion, etc.). To pursue these objectives, extensive data collection and experimentation will be performed with a number of image capture designs and methods for detecting and analyzing discontinuities. The research will enable a variety of both short-term and long-term applications in engineering, medicine, art, and entertainment. Real and synthetic data for experiments will be shared publicly with the research community via the web, as will metrics and software for comparing and evaluating techniques. The educational impacts of the project include the involvement of graduate and undergraduate students and new seminar courses that will be developed related to the research theme. The project will address diversity issues by recruiting undergraduate researchers and short-term graduate students through programs that aim to increase the involvement of underrepresented students in science and engineering, and also by the planned collaborations with colleagues and students in areas outside of Computer Science where women are currently better represented.

In this research project, the PI proposes to develop novel methodology to visualize and interact with large interconnected high-dimensional datasets represented as large unstructured graphs. The underlying semantics for the nodes and edges in such a graph are to be kept flexible, but the ultimate goal is for an analyst to acquire an understanding of the data universe as a whole, and to enable him or her to issue and resolve specific data queries. An analyst should also have an easy way to inform a colleague about the insights gathered during their own interaction with the system, including evidence chains and new hypotheses.

Proposal #: CNS 08-21858
PI(s): Turk, Matthew
Hollerer, Tobias H.; Hu, Evelyn L.; Kuchera-Morin, JoAnn C.; Wolski, Richard
Institution: University of California ? Santa Barbara
Santa Barbara, CA 93106-2050
Title: MRI/Dev.: Dev. of the Allosphere, an Immersive Instrument for Scientific Exploration
Project Proposed:
This project, contributing in the development of an advanced multimedia immersive environment and scientific instrument for large-scale data visualization and exploration, aims to provide powerful methods for detail analysis, synthesis, and manipulation of complex multi-dimensional data where the observed phenomena range from the very big to the very small. The instrument integrates the multimodal representation of large-scale data with human-scale visualization and interaction techniques in a novel immersive environment known as the Allosphere. The Allosphere is a three-story cubic space in the new California NanoSystems Institute, housing a large perforated metal sphere that serves as a display surface. A bridge that can hold 25 participants cuts through the center of the sphere from the second floor. The final Allosphere instrumentation design is expected to include 14 resolution stereo video projectors to light up the complete spherical display surface, 256 speakers distributed outside to provide high quality spatial sounds, a suite of sensors and interaction devices to enable rich user interaction with the data and simulations, and the computing infrastructure to enable the high volume computations necessary to provide a rich visual, aural, and interactive experience for the user. When fully constructed, the Allosphere is expected to be one of the largest scientific instruments in the world. It will serve as an ongoing research testbed for scientific visualization, numerical simulations, large-scale sensor networks, high-performance computing, data mining, knowledge discovery, multimedia systems, and human-computer interaction. This unique immersive exploration environment, with a full surrounding sphere of high quality stereo video and spatial sound and user tracking for rich interaction will support rich interaction and exploration. Partnering with faculty from several disciplines, it will also serve as a key tool for exploration in molecular dynamics, brain imaging, nanomaterials, earth science, and many other scientific fields. For example, navigating through fMRI data and exploring live geospatial datasets.

Broader Impacts: This project benefits many important fields that have wide impact on society. The Allosphere will be made available to researchers and partners beyond the institution and will leverage local outreach programs to attract K-12 students involving them early in scientific exploration projects that utilize the environment. The Allosphere is likely to draw students of all ages and interests.

The activity of visualizing and exploring complex multi-dimensional data provides insight that is essential for progress in several areas of science and engineering, where the amount and complexity of the data overwhelms traditional computing environments. This project will develop the first complete version of the Allosphere, an infrastructure that will provide powerful methods for detailed analysis, synthesis, and manipulation of such data by integrating multimodal representations of large-scale data with human-scale visualization and interaction techniques in a novel immersive environment.

Intellectual Merit: The Allosphere comprises a 10m diameter spherical display surface in a three-story near-anechoic building space, with a bridge running through the center that holds up to 25 participants. When fully equipped, the Allosphere will include high-resolution stereo video projectors to illuminate the complete spherical display surface, a large array of speakers distributed outside the surface to provide high quality spatial sound, a suite of sensors and interaction devices to enable rich user interaction with the data and simulations, and the computing infrastructure to enable the high-volume computations necessary to provide a rich visual, aural, and interactive experience for the users. The Allosphere will be one of the largest visualization/exploration instruments in the world, and it will serve as an ongoing research testbed for several important areas of computing. In addition, the Allosphere will serve as an environment for experimental media creation and performance, as a tool for scientific discovery in areas such as nanosystems, neuroscience, quantum computing, and biochemistry, and as an instrument for education and outreach.

Broader Impacts: This infrastructure project will complete the audio, interaction, and computational infrastructure of the Allosphere. This first version will serve as a computing research infrastructure in two primary ways: (1) as a platform for driving computing research needed to create future immersive multimodal, multimedia visualization environments, raising challenging problems in storage, networking, rendering, software virtualization, real-time simulation, and human-computer interaction; and (2) as an immersive visualization environment for research in computing areas such as scientific and information visualization, knowledge discovery, visual analytics, complex design, large-scale performance debugging, data mining, and cloud computing. As infrastructure for both computing and scientific exploration, the Allosphere will benefit a number of important fields that have wide impact on society. The facility will be made available to researchers and partners (in academia, industry, and government) beyond UCSB. The project will also leverage local outreach programs to attract K-12 students, underrepresented groups, and undergraduate students, and involve them in scientific projects and explorations that utilize this unique immersive environment. The facility will be integrated into courses and research projects in several departments at UCSB. Results will be disseminated via the project Web site (http://allosphere.ucsb.edu).

The goal of this project is to develop and evaluate novel methods for telecollaboration - remote collaboration that effectively integrates video and voice communication, computer vision based tracking, and augmented reality display in order to enable participants to more fully leverage the local physical environment in their remote interactions. In this telecollaboration paradigm, remote and local users will interact with the physical environment using models derived from live imagery from a camera that the local user holds or wears, rather than merely passively viewing the video stream. A key aspect of the approach is that it does not require preparation of the environment or model information, so it can be viewed as an "anywhere, anytime" mobile communication technology.

Intellectual merit: The proposed research builds on promising preliminary work on telecollaboration showing the effectiveness of world-stabilized "telepointers" - markers controlled by the remote user that stick to the real-world referents in a dynamic environment. The project will advance the state of the art in remote collaboration by providing new capabilities to integrate real and virtual, verbal and spatial, local and remote. The proposed developments are needed in order to make the physical space a more fundamental part of telecollaboration, and significant user studies will be conducted to acquire a better understanding of the opportunities and limitations of physically-grounded remote interaction.

Broader impacts: Better, more compelling tools for telecollaboration can have a tremendous impact in terms of productivity and environmental consequences, as improved remote interaction reduces the need for physical collocation and thus travel. The educational impacts of the project include the training and mentoring of graduate students and a new seminar course. The research team will disseminate research results and collected data widely and use the developed technologies to provide outreach opportunities for select groups to participate in lab open houses.

Computational modeling of astrophysical phenomena has grown in sophistication and realism, leading to a diversity of complex simulation platforms, each utilizing its own mechanism and format for representing particles and fluids. Similarly, most of the data analysis is conducted with tools developed in isolation and targeted to a specific simulation platform or research domain; very little systematic and direct technology transfer between astrophysical researchers exists. The yt project is a parallel analysis and visualization toolkit designed to support a collaborative community of researchers as they focus on answering physical questions, rather than the technical mechanics of reading, processing and visualizing data formats. This project will enable the development of advanced, physics-based modules that apply universally across simulation codes, advancing scientific inquiry and enabling more efficient utilization of computational and human resources. In doing so, it will help advance a myriad of research goals from the study of black hole binaries to the growth of cosmic structure. In addition, the project will serve as a touchstone for collaboration and cross-code utilization between many groups studying diverse phenomena. Moreover, the project will be developed through a community-oriented process, engaging a wide range of participants.

The infrastructure development in this research will enable these capabilities by broadening the applicable simulation platforms within yt, enabling cross-code utilization of microphysical solvers and physics modules and in situ analysis, and developing collaborative platforms for the exploration of astrophysical datasets. In particular, it will develop the capabilities of yt in three primary mechanisms. The first is to enable support for additional, fundamentally different simulation platforms such as smoothed particle hydrodynamics, unstructured mesh, and non-Cartesian coordinate systems. The second is to provide simulation instrumentation components to ease the process of developing simulation codes, interfacing and exchanging technology between those simulation codes, and to enable deeper, on-the-fly integration of astrophysical simulation codes with yt and other analysis toolkits. The final focus is on developing interface components to enable collaborative and interactive exploration of data utilizing web-based platforms. An explicit goal of this SI2-SSE project is the development of collaborative relationships between scientists, furthering the development of the field as a whole. By conducting all business in the open with a focus on developing and encouraging collaborative, welcoming environments for contributors and researchers, this SSE will help to foster a level playing field that is more accessible to all parties, particularly women and underrepresented minorities. An explicit milestone of this project is to streamline the process of conducting direct outreach through scientific visualization, greatly expanding the domains and individuals engaged in STEM-based public outreach.

Educational experiences for students that focus on online interactions are becoming increasingly important in current educational contexts. New technologies for video communication and technologies expand the pedagogical contexts well beyond the typical video chat applications and have the potential to enhance both instructor-student and peer-to-peer models of teaching. This project is a new collaboration between researchers at the University of California at Santa Barbara and two universities in Finland to explore the development and use of transformed social interaction and tele-collaboration technologies in the context of learning and education. The focus of the collaboration is to create a new community of scholars, identify key research questions and relevant issues, share particular expertise and experience related to the effort and plan for a collaboration with well-articulated research objectives.

This project will involve a series of meetings that will catalyze the new partnership in research on collaborative learning technologies. The researchers from the US will include two graduate and one undergraduate student from computer science and media arts and technology and two graduate and one undergraduate student from education. The Finnish universities will include several students and researchers in addition to the core faculty involved. A one-week conference in Finland will be proceeded with sharing of information and publications among the groups. Follow-up teleconferencing meetings will provide the opportunities to continue to refine the outcomes and develop project descriptions of potential research projects.

This project enables global-scale dynamic reconstructions that can scale to eventually encompass all of the world's 3D data, to which any user may contribute new visual data, thereby ensuring a more complete, up-to-date model and a better experience for users of applications that rely on such data. The abundance of publicly available imagery from a variety of sources (consumer, industry, and government) and the proliferation of networked mobile devices equipped with cameras provide an opportunity to build large-scale 3D models that cover the entire world. Such global 3D models are invaluable to a wide range of applications that require real-time access to environment structure, such as providing assistance to the visually impaired, exploring dangerous areas for search-and rescue operations, urban planning, self-driving cars, and virtual tourism.

The key research question driving this work is how to efficiently and accurately update a global-scale 3D reconstruction in order to support a dynamic global model. The research uses structure from motion (SfM) techniques to acquire global context for 3D modeling and then propagate local 3D reconstructions and other visual data back to the global model via a new technique called "globalization." The project addresses a major gap in large-scale SfM: rapidly extending and updating reconstructions to promote real-time use. The project is built on preliminary work in large-scale image matching, localization, and 3D reconstruction. The research team carries out extensive data collection and experimentation to benchmark the performance of the developed techniques and to assess the progress of the project and the utility of the methods. The project is developing a prototype mobile system for contributing imagery and browsing and updating models for public use once it is available.

This project will host a 1.5-day workshop in Arlington, VA, which will bring together the community of SI2 awardees (with the goal of involving one principal investigator from each SSE and SSI project, many of which are collaborative awards) from 214 awards. The workshop will also solicit participation from NSF-supported science and engineering projects that rely on the national research cyberinfrastructure for their computation, communication, and data management needs. In addition, the proximity to NSF will encourage participation by Program Officers from across the Foundation. Goals of this workshop include: (1) Providing a focused forum for PIs to share technical information with each other and with NSF Program Officers; (2) Encouraging exploration of emerging topics; (3) Identifying emerging best practices across the supported software projects; (4) Stimulating thinking on new ways of achieving software sustainability; and (5) Disseminating the shared experiences of the researchers via an online web portal. The workshop is expected to host 85 SI2 awardees and several other speakers and panelists. The workshop will use a hybrid style, blending a traditional, top-down driven agenda with a interactive, adaptive, participant-driven agenda.

The proposed workshop will support the exchange of ideas among the current software cyberinfrastructure development projects. It will provide guidance on issues related to the development of robust software and to the problem of software sustainability. Involvement of program officers across NSF is expected to help the interdisciplinary SI2 awardees understand the relevance and impact of cyberinfrastructure throughout the NSF. The participation of these researchers and program officers in a common forum will help ensure that the cyberinfrastructure software developed as part of SI2 projects will be relevant and broadly applicable to the most science and engineering domains possible. The hybrid approach is innovative and holds the promise of creating rich interactions among PIs, resulting in richer collaboration and learning. The results of this workshop thus have the potential to guide cyberinfrastructure development and cyberinfrastructure driven research for both the participating projects and for the wider software development community.

Part 1

Changes of a galaxy's properties over time are driven by the quantity of its cold gas, the raw material from which stars form. Thus, understanding the properties of a galaxy's cold gas component will tell us both how the star formation process changes over time and how this affects galaxies. Using a recently completed survey of carbon monoxide gas in a sample of nearby galaxies, the proposers will (1) measure how the molecular gas is used up, in order to test how star formation in the galaxies is affected; (2) map the locations of this gas in the galaxies to determine if/how these locations affect star formation; and (3) measure how the gas is flowing into and out of the galaxies to determine how this affects structural changes in the galaxies over time. In particular, some questions that can be answered from this research include: What factors affect the conversion of molecular gas into stars? How is molecular gas structured? How do galaxies grow and age? This proposal will provide insight into the process of galaxy formation and our understanding of these objects.


Part 2

This proposal plans to combine spatially resolved spectroscopy of carbon monoxide molecular gas, ionized gas and stellar populations for a sample of 125 nearby galaxies. Measuring spatially-resolved molecular gas depletion time scales as a function of various galaxy properties, using diagnostics of spectral lines to probe diffuse molecular gas, and constraining the rates of gas in- and outflow for these galaxies has the potential to add greatly to our understanding of star formation in normal galaxies.

Observations of galaxies across a range of redshifts allow us to witness the growth and aging of the galaxy population. It has long been recognized that these processes are driven by the rise and fall of star formation, which is ultimately linked to the supply of cold gas. Further progress in understanding the regulation of star formation in galaxies, however, requires additional observational constraints beyond the molecular gas mass and star formation rate. By exploiting a unique combination of the largest, most homogeneous interferometric CO survey of galaxies to date (the Extragalactic Database for Galaxy Evolution, or EDGE) and one of the principal ongoing optical integral field unit (IFU) surveys (CALIFA), the project team will confront models of star formation and galaxy evolution with uniform, high quality measurements of gas masses and star formation rates, supplemented by a vast array of additional data, including star formation history, stellar mass, nuclear activity, and the metallicity and kinematics of the gas and stars. These studies will be conducted on a spatially resolved basis across a well-defined sample that is representative of the nearby Universe, and will thus inform future studies of star formation and mass assembly at all redshifts undertaken by JWST and ALMA.

A great challenge in astrophysics is to understand in detail how the initial smooth distribution of matter in the early Universe formed the first galaxies. Complementing observations of real galaxies, researchers use computational simulations to model the early Universe and study the results. This process allows one to learn how these first galaxies might have formed. However, the sheer size and complexity of such galaxy simulations present their own challenge as a single research group lacks the capacity to explore them fully. As a result, maximizing the scientific value of simulations demands new tools and services designed to foster the growth of a collaborative, multi-group research community. This project aims to develop and use a new virtual laboratory to enable transformative scientific inquiry on new and existing galaxy simulations, some of which were produced by prior NSF support. Enabling public access and unrestricted analysis and fostering a collaborative environment for sharing technology and results will ensure that galaxy simulations continue to be valuable within and beyond the research group that originally conducted them. This project addresses the national imperative to develop US cyber infrastructure and to develop US leadership in scientific research in astrophysics.

More technically, with prior NSF support the investigators used the Blue Waters supercomputer at the National Center for Supercomputing Applications (NCSA) to perform the Renaissance Simulations: among the largest, most detailed simulations of the formation and evolution of the first galaxies to date. The Renaissance Simulations yielded a sample of over 3,000 high redshift (20 > z > 7) galaxies and were specifically designed to simulate the dominant sources responsible for preheating the intergalactic medium and reionization at an unprecedented level of detail. This project plans two related activities: (1) further analysis of the Renaissance Simulations; and (2) the creation of an open-data access portal, the Renaissance Simulation Laboratory (RSL), which will provide open access to this unique data corpus along with associated data analysis tools for the astronomical research community. Driving its design and utility, the investigators will use the RSL to carry out their own research investigations. The investigators' research processes will become part of the RSL in the form of executable "Jupyter" notebooks. The Jupyter Notebook, an evolution of the iPython Notebook, is a fundamental part of the RSL and is ushering in the era of open science. The investigators' use of Docker containers is equally compelling. By containerizing Jupyter Notebooks which execute pre-programmed data analysis workflows the investigators are able to bring computation to the data and share how they obtain their scientific results, a key step toward reproducible science. Users will have the option of downloading notebooks and associated data to their own platforms or executing them on San Diego Supercomputer Center and NCSA high performance platforms.

Scholarly publications today are still mostly disconnected from the underlying data and code used to produce the published results and findings, despite an increasing recognition of the need to share all aspects of the research process. As data become more open and transportable, a second layer of research output has emerged, linking research publications to the associated data, possibly along with its provenance. This trend is rapidly followed by a new third layer: communicating the process of inquiry itself by sharing a complete computational narrative that links method descriptions with executable code and data, thereby introducing a new era of reproducible science and accelerated knowledge discovery. In the Whole Tale (WT) project, all of these components are linked and accessible from scholarly publications. The third layer is broad, encompassing numerous research communities through science pathways (e.g., in astronomy, life and earth sciences, materials science, social science), and deep, using interconnected cyberinfrastructure pathways and shared technologies.

The goal of this project is to strengthen the second layer of research output, and to build a robust third layer that integrates all parts of the story, conveying the holistic experience of reproducible scientific inquiry by (1) exposing existing cyberinfrastructure through popular frontends, e.g., digital notebooks (IPython, Jupyter), traditional scripting environments, and workflow systems; (2) developing the necessary 'software glue' for seamless access to different backend capabilities, including from DataNet federations and Data Infrastructure Building Blocks (DIBBs) projects; and (3) enhancing the complete data-to-publication lifecycle by empowering scientists to create computational narratives in their usual programming environments, enhanced with new capabilities from the underlying cyberinfrastructure (e.g., identity management, advanced data access and provenance APIs, and Digital Object Identifier-based data publications). The technologies and interfaces will be developed and stress-tested using a diverse set of data types, technical frameworks, and early adopters across a range of science domains.

This project will host a 1.5-day workshop in Arlington, VA, which will bring together the community of SI2 awardees (with the goal of involving one principal investigator from each SSE and SSI project,many of which are collaborative awards) from approximately 250 awards. The workshop will have participation from CDS&E, CRISP and Venture funded PIs as well as SI2 EAGER and RAPID awardees and selected awardees of the ACI VOSS program that examines cyberinfrastructures from the social and organizational perspective. In addition, the proximity to NSF will encourage participation by Program Officers from across the Foundation. Goals of this workshop include: (a) providing a focused forum for PIs to share technical information with each other and with NSF Program Officers, (b) encouraging exploration of emerging topics, (c) identifying emerging best practices across the supported software projects, (d) stimulating thinking on new ways of achieving software sustainability, and (d) disseminating the shared experiences of the researchers via an online web portal.

The workshop is expected to host close to 150 SI2 and other awardees, other speakers and panelists. The workshop will use a hybrid style, blending a traditional, top-down driven agenda with a interactive, adaptive, participant-driven agenda.

The proposed workshop will support the exchange of ideas among the current software cyberinfrastructure development projects. It will provide guidance on issues related to the development of robust software and to the problem of software sustainability. Involvement of program officers across NSF is expected to help the interdisciplinary SI2 awardees understand the relevance and impact of cyberinfrastructure throughout the NSF. The participation of these researchers and program officers in a common forum will help ensure that the cyberinfrastructure software developed as part of SI2 projects will be relevant and broadly applicable to the most science and engineering domains possible. The hybrid approach is innovative and holds the promise of creating rich interactions among PIs, resulting in richer collaboration and learning. The results of this workshop thus have the potential to guide cyberinfrastructure development and cyberinfrastructure driven research for both the participating projects and for the wider software development community.

A new astronomy has arrived with the recent detection of gravitational waves. Modeling of sources of gravitational radiation is more than ever a critical necessity in order to interpret the observations. The project Einstein Toolkit has as overarching mission to provide the scientific community with a sustainable software platform of core computational tools for research focused on astrophysical systems endowed with complex multi-scale/multi-physics properties which are governed by Einstein's equations of General Relativity. The central premise of the project Einstein Toolkit is to create a broad and vibrant community of users, a community where interdisciplinary collaborations are the norm and not the exception, a community driving advances in the next generation of high-performance computing cyberinfrastructure. The main objectives of the project Einstein Toolkit are: developing software tools for a radical increase in scientific productivity, achieving sustainability of the software ecosystem, addressing software engineering challenges, and the curation of data from general relativistic numerical simulations.

This project will achieve its goals through two major activity areas. Regarding the software ecosystem and its sustainability, the scheduler that handles the flow of tasks in a problem will be redesigned to be more versatile and to improve its performance. In addition new software modules will be developed to broaden the choices of initial data and matter sources, as well as modules for problems requiring a high degree of experimentation with equations and numerical methodologies. A new general relativistic magneto-hydrodynamics code will also be integrated in the Einstein Toolkit. The second activity area involves building a simulation data repository. The repository will allows user to compare results, contribute data, test innovative ideas and algorithms for gravitational wave data analysis, and to explore or discover new phenomena in sources of gravitational radiation. The broader impact effort in the project Einstein Toolkit will be organized in two major activity areas. The first involves community integration. The project will support a program of ease-of-use on-line tutorials and a workshops/tutorial series. The program will help small groups or individual investigators familiarizing with the codes and modules in the toolkit as well as pathways to become a developer. Regarding outreach and education, the project Einstein Toolkit will enable interdisciplinary training of students and postdocs in numerical relativity, computational astrophysics and computer science. The effort will includes developing a teaching resources bank for educational activities involving computational topics applied to gravitational physics and astrophysics. The educational resources will be suitable for computational courses in general relativity and astrophysics at both the graduate and undergraduate level.

This project is supported by the Division of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science & Engineering and the Physics Division in the Directorate of Mathematical and Physical Sciences.

This project will host a 1.5-day workshop in Arlington, VA, which will bring together the community of SI2 awardees (with the goal of involving one principal investigator from each SSE and SSI project, many of which are collaborative awards) from approximately 250 awards. The workshop will have participation from SI2 EAGER and RAPID awardees and selected awardees of the ACI VOSS program that examines cyberinfrastructures from the social and organizational perspective. In addition, the proximity to NSF will encourage participation by Program Officers from across the Foundation. Goals of this workshop include: (a) providing a focused forum for PIs to share technical information with each other and with NSF Program Officers, (b) encouraging exploration of emerging topics, (c) identifying emerging best practices across the supported software projects, (d) stimulating thinking on new ways of achieving software sustainability, and (d) disseminating the shared experiences of the researchers via an online web portal.

The workshop is expected to host close to 150 SI2 and other awardees, other speakers and panelists. The workshop will use a hybrid style, blending a traditional, top-down driven agenda with a interactive, adaptive, participant-driven agenda.

The proposed workshop will support the exchange of ideas among the current software cyberinfrastructure development projects. It will provide guidance on issues related to the development of robust software and to the problem of software sustainability. Involvement of program officers across NSF is expected to help the interdisciplinary SI2 awardees understand the relevance and impact of cyberinfrastructure throughout the NSF. The participation of these researchers and program officers in a common forum will help ensure that the cyberinfrastructure software developed as part of SI2 projects will be relevant and broadly applicable to the most science and engineering domains possible. The hybrid approach is innovative and holds the promise of creating rich interactions among PIs, resulting in richer collaboration and learning. The results of this workshop thus have the potential to guide cyberinfrastructure development and cyberinfrastructure driven research for both the participating projects and for the wider software development community.

Scientific discovery across the physical sciences is increasingly dependent on the analysis of volumetric - or three-dimensional - data, that may come from a supercomputer simulation, direct measurement, or mathematical models. Researchers typically seek to extract meaningful insights from this data by visualizing and analyzing it in various ways. The ways in which scientists process volumetric data are actually quite similar across domains, but cross-disciplinary knowledge transfer and tool development is blocked by barriers of terminology. This project seeks to enhance an analysis and visualization toolkit named yt that is currently primarily used for astrophysical simulations. yt allows scientists to access and analyze data at several different levels by providing an interface that is designed to answer questions motivated by the underlying scientific problem, while worrying less about details such data formats, specific analysis techniques etc. yt's utilization in computational astrophysics has dramatically increased access to advanced algorithms for both visualization and analysis, and fostered the growth of a community of researchers sharing techniques and results. This project seeks to make yt available and adopted by scientists in other domains of science thus reproducing its success in astrophysics in these other science domains. This project will expand the yt community beyond theoretical astrophysics and enable and promote collaboration and advanced data analysis in the fields of meteorology, seismology and global tomography, observational astronomy, hydrology and oceanography, and plasma physics.

Improvements to the yt project will proceed along four principal technical avenues. The first is to develop a system that adapts the way yt presents data via a set of domain contexts that encode the ontology, domain-specific vocabulary, and common analysis tasks for a given field of study. This will include creating a domain context system as well as a set of five pilot domain contexts developed in collaboration with domain practitioners. The second is to overhaul the yt field system, adding more versatility and enabling significant optimizations. Thirdly, the project team will implement non-spatial indexing schemes, providing methods for accessing and analyzing data that is not organized according to the standard spatial axes. The final improvement will be the development of a non-local analysis system, allowing generalized path traversal as well as domain convolutions. To ensure wide dissemination and use of these improved capabilities, the team will design domain-specific documentation and training materials, and organize outreach and training events for early-career researchers. This will consist of both hands-on technical workshops and curricula developed in collaboration with Data Carpentry for utilization at other institutions. This combination of technical developments and social investments has been designed to ensure both readiness of the software and engagement of the targeted research communities.

This project develops new technologies that measure and model people's state of attention and applies these methodologies to virtual reality (VR) and augmented reality (AR) language learning applications. For determining the degree of people's attention on central learning tasks presented in VR and AR, this project uses two sensor modalities that can index attention: electrical activity of the brain, as measured by electroencephalography (EEG) signals, and eye gaze behavior, as measured by eye trackers. Context recognition plays a key element in future VR and AR application scenarios, and users, as well as content providers, can benefit substantially from information about user attention states during information consumption. The project can inform the development of optimized VR and AR content, as well as individualized learning strategies. The project's motivating application is the optimization of language learning for users across the complete spectrum of ability. In the longer run, additional benefits include the creation of special tools for students with known attention deficits, as well as for increasing productivity and safety in various commercial and industrial applications.

This research explores necessary novel technologies for attention-aware mixed reality (MR) interfaces. The project integrates signals from consumer-grade EEG and eye tracking devices, to determine if and how much the participant's attention is divided (i.e. distracted or multi-tasking) or focused, and to assist appropriately. In this attention-assisted paradigm, users are monitored by EEG and eye tracking devices while interacting with mixed reality user interfaces. Attention states are classified over time using both sensor modalities and can be spatially referenced in the user interface with eye tracking. Attention activity feedback can be reported in real time while the user is interacting with the interface or may be stored and later visualized for a more thorough analysis of attentional patterns. The project delivers technology demonstrations of attention-aware interfaces for foreign language vocabulary learning in VR (with a lookout on AR possibilities). Exploratory experiments will uncover the possibilities for characterizing human attention states through EEG and eye tracking data. The project opens new opportunities for advances in multi-modal interaction to contribute to MR interfaces and learning technologies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The application of engineering materials impacts every facet of society, through manufacturing, energy, transportation, medicine, and more. One the most important trends in materials science is the emergence of atomic scale characterization of materials. Characterizing and modeling materials at the atomic scale generates massive data sets. New advances in data science are now revolutionizing the way that materials data are captured, curated, managed, and manipulated. Industry and national labs are increasingly in need of a science and engineering workforce trained in both materials and data science. This National Science Foundation Research Traineeship (NRT) award to the University of Illinois will support the creation of a Data and Informatics Graduate Intern-Traineeship in Materials at the Atomic Scale (DIGI-MAT). The vision of DIGI-MAT is that materials problems will ultimately be data problems, and understanding of materials will be a challenge in capturing, curating, managing, and manipulating massive data streams. The program will combine cutting-edge research with a new curriculum, professional development opportunities, on-demand skills training, and, at the core of the program, research internships with external partners. This project anticipates training 72 PhD students, including 31 funded trainees, from degree programs in engineering, physics, statistics, and information science.

The DIGI-MAT project balances research with coursework, skill-building, professional development and robust internships. Working across disciplinary boundaries, participants will tackle specific, high-impact research problems, including the following: (i) machine-learning to identify features in billion-atom microscope images; (ii) statistics and data-fusion methods for large materials data sets that vary over space and in time; (iii) new routes to automate the synthesis of nanomaterials using data; (iv) cyberinfrastructure for atomistic data; and (v) new highly accurate and efficient methods to computationally model materials. The program will integrate this research activity with new interdisciplinary graduate course offerings. Additional flexible training modules will be offered in data acquisition and management, materials data curation, uncertainty quantification, instrumentation for materials characterization, and application of machine learning to materials data. Trainees will be placed in semester-to-year-long research internships with partners in industry, national labs, academia, and abroad. Research internships will be closely aligned with trainees' graduate research, helping to prepare them for careers at the intersection of materials and data science. Finally, trainees will have the opportunity to develop additional career-aligned skillsets such as communication, leadership, and entrepreneurship through workshops and other venues (e.g. industry networking events).

The NSF Research Traineeship (NRT) Program is designed to encourage the development and implementation of bold, new potentially transformative models for STEM graduate education training. The program is dedicated to effective training of STEM graduate students in high priority interdisciplinary or convergent research areas through comprehensive traineeship models that are innovative, evidence-based, and aligned with changing workforce and research needs.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Augmented reality (AR) is a technology that superimposes computer-generated visual, auditory, or other sensory information onto the physical world, so that they appear to be part of the actual environment. The goal of the proposed project is to develop and evaluate new methods for creating, maintaining, and improving large-scale scene models to enable wide-area AR. For example, consider an AR application for firefighters, which superimposes predicted changes in fire perimeter and coordinated action plans to help them understand weather impact and communicate the current plan of attack. Augmented reality is characterized by the fact that these visualizations are three-dimensionally registered with the real world, keeping them matched to the physical environment in real-time even as the user moves around their environment. In order to place new virtual annotations (and to keep track of previously placed annotations), a three dimensional "scene model" that represents the physical world locations needs to be created. Ideally, this would be done on a global, world-wide scale to include all possible places where AR experiences are possible or have even already occurred. Since it is difficult for one person or organization to collect the data needed for such a wide-area scene, the developed system will aggregate crowd-sourced data of different modalities (e.g., images, videos, and 3D geometrical meshes) from multiple sources, leveraging semantic information in addition to basic image and point cloud data. By providing capabilities for remotely guiding a local AR user to capture new imagery or sensor data to achieve more accurate or complete models, crowd-sourced modeling can be directed over time to create and continuously update scene models of large-scale areas, such as a university campus or even a city. The research will result in a system that effectively integrates multiple components to provide new opportunities to remotely navigate, explore, and augment physical spaces for AR applications in government, education, industry, and consumer spaces.

To accomplish the above objectives, the researchers will implement and utilize a Dynamic Hybrid Scene Model Server that accepts crowd-sourced image, video, and point cloud data, and continuously performs smart data integration and completion, leveraging machine learning approaches to infer semantic information from the raw image and point cloud data and to fill in missing information. The main challenge here will be to design the learning approaches in such a way that it will not require access to all the low-level data fusion and filtering components, simply because the information may not be available in the crowd-sourced data. Augmented Reality and Virtual Reality user interfaces will be developed and evaluated to deal with imperfect and incomplete environment models produced by the server, allowing remote users to virtually navigate through modeled spaces and to provide guidance to the local AR users. On the human interface side, the project focuses on research in remotely navigating and exploring "visual reality", virtual models of the real-world spaces created from the crowd-sourced imagery and creating content for augmenting the visual reality. The proposed methods will address limitations in current mixed reality applications in modeling and remote navigation, providing users an experience of remote visual reality that is valuable as a replacement for physical navigation, as a training aid for planned activity, and as a way to share information in augmented and virtual reality environments. The project will leverage existing work by the team of researchers and others on image-based modeling, virtual scene navigation, and virtual annotation for remote collaboration, and it focuses on both the key system components and the user experience to explicitly support navigation and augmentation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Modern laboratories provide unprecedented sensitivity to the many different galactic-messengers that stream through our planet by the minute: cosmic rays, light from distant galaxies, elusive neutrinos, and possibly dark matter. Combining this information with models and data from simulations provides insight into how our universe began and continues to evolve -- the scales at which objects first collapsed, the development of stars and galaxies, and the dynamics within our own galaxy.

However, this data is often inaccessible: scientists within an experiment or community struggle with the complex, custom-built programs they use to access the data. And switching to a standard format is usually not an option: these data formats are designed for requirements that often do not include cross-experiment synthesis or linking.

Junior scientists - let alone the public - can struggle to generate new insights from the data because the data is difficult to access, understand and analyze. The cross-cutting inquiry that could arise from clever reuse and combination of data from different experiments and simulations is rarely conducted.

This project makes data accessible both within and across collaborations, providing the infrastructure to search for signals in detectors across the globe. Extending existing efforts to improve data access makes this project possible: yt is software that provides uniform access to simulation data; Kaitai is a data-description language that enables easy access to any data format; Rucio and other tools provide a standard interface that allows data downloads; and ServiceX can identify, subset and process data with little effort from the end user.

Scientists have built experiments that offer an incredible wealth of information about our world. This project works to make that information accessible to everyone.

Technical Description

The Personal Data-Delivery infrastructure (PONDD) addresses the data challenges of existing dark matter and astrophysics experiments while requiring no changes to existing data formats. This non-invasive, no-changes-necessary support for any file format provides opportunities to expand beyond our two identified use cases, dark matter searches and astrophysics simulations, into many other data-driven science domains that rely on custom file formats.

This work delivers an infrastructure that seamlessly delivers data in a well-supported format (such as Parquet) from multiple sources. To successfully deliver cross-experiment data to end users, we bring together ongoing projects from High Energy Physics and the broader NSF community; while this project will involve development of software products (yt and Kaitai) it will also include synthesis of existing investments in cyberinfrastructure and efforts to improve their long-term sustainability.


This project is supported by the Office of Advanced Infrastructure in the Directorate for Computer and Information Science and Engineering and the Division of Physics in the Directorate for Mathematical and Physical Sciences.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The Earth’s mantle, which sits directly below the crust, is predominantly made of solid rock; yet the solid mantle can flow when pushed or pulled. The rate of this flow depends on the properties of the rock, such as its temperature, and on the nature of the contacts between the tiny mineral crystals that comprise the rock. The mantle can be pushed to flow by numerous different phenomena, such as: passing seismic waves after an earthquake; melting of continental ice sheets and glaciers; the annual cycle of groundwater recharge and extraction; and the draining of large lakes. This study uses observations of these phenomena to measure the rock properties and the interactions between mineral crystals in the mantle beneath three locations: the western United States, Alaska, and Iceland. Meanwhile, laboratory experiments are probing how samples of rock deform under controlled conditions. Finally, new computer models are synthesizing the lab and field observations to understand the underlying physical laws that explain the full suite of data. The results of this study have a bearing on topics that range from predicting how sea level will rise due to melting ice sheets to understanding tidal deformation on Jupiter’s moons. Outreach and training are key elements of the project. Four graduate students and six undergraduate students are being educated over the duration of the project. Workshops will bring together researchers from diverse scientific disciplines to learn and debate about the scientific outcomes and the computer tools developed as part of this study.<br/><br/>There is emerging recognition that the variables describing Earth’s mechanical response to stress, elastic moduli, attenuation, and viscosity, are all frequency dependent. While the end-member elastic and steady-state behaviors are relatively well understood, there remain many fundamental questions regarding the intermediate transient regime. This study is an integrative research and outreach program that combines observational, laboratory, and modeling efforts to measure Earth’s full-spectrum rheological response and illuminate the underlying microphysical processes. Observational work is characterizing frequency dependent upper-mantle dissipation in three locations (western U.S., Iceland, and Alaska) using seismic and geodetic observations of different frequencies but complementary spatial sampling. Experimental work is investigating how dislocations affect transient creep under different temperature and stress conditions and with variable quantities of melt and secondary solid phases. Modeling work is developing new constitutive laws for transient creep and incorporating more sophisticated rheologies in the viscoelastic deformation code. This study is addressing questions about: (1) the broadband mechanical response of the solid Earth; (2) the microphysical processes that control viscoelasticity; and (3) the implications for inferences of steady-state viscosity from geodetic observations and of thermodynamic state from seismic tomography. Broader impacts include training of graduate and undergraduate students, a synthesis workshop that convenes 120 researchers to outline recent advances in understanding transient rheology and to shape the topics and collaborations that will dictate the next decade of inquiry, and development of interactive Jupyter notebooks that introduce open-source data-science tools in the context of seismic attenuation and transient rheology.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.