Bilge Mutlu, Ph.D. - US grants

The goal of this project is to design and build gaze mechanisms for embodied agents that can achieve high-level social and communicative goals that people achieve using such mechanisms and that can be applied across a wide range of agent presentations and task domains.

Embodied agents promise significant social, cognitive, and organizational benefits through applications in education, training, rehabilitation, and collaborative work. However, in order to be effective across a wide range of applications, agents must be able to use the nonverbal social cues that humans employ in their communication and to employ these cues in whatever modality the agent is presented in - whether it be a social robot, a life-sized virtual human, or an animated avatar on a portable display. Gaze cues are particularly important social signals. Although they are subtle, they can serve as powerful mechanisms for achieving high-level social and communicative goals, such as improving a listener?s comprehension, controlling the flow of a conversation, and indicating interest in or appraisal of objects. This project investigates how such mechanisms might be designed and built for embodied agents and how similar social and communicative goals could be achieved using different agent representations across different task domains.

This investigation will involve (1) performing formal observational studies to better understand how people use gaze, (2) developing computational models that synthesize gaze behaviors that can be controlled precisely and retargeted to a range of agent platforms, and (3) evaluating in experimental studies the effectiveness of using gaze cues across a range of agent presentations and task contexts. Success in this project will create new knowledge on human gaze behaviors, connecting the high-level findings in the social science literature to more detailed, low-level cues and mechanisms. It will also produce a set of techniques that are based on this understanding for synthesizing controllable and flexible gaze movements that agents can use in order to achieve social and communicative goals. Finally, it will validate the effectiveness of the use of gaze cues by agents across a variety of agent presentations and task contexts.

A trans-disciplinary approach will combine rigorous, formal observational studies to build detailed models of human communicative mechanisms with practical efforts to build usable computational models that meet the needs of creating agents that work in real-world tasks. The focus on human models insures that the computational models are well founded, while the focus on developing practical algorithms guides the human studies towards creating understanding that will be most informative for agent design. The project plan involves connecting the disparate communities that work on developing social agents and training students to do the trans-disciplinary work required to create effective embodied agents. The project will also enable K-12 outreach efforts to use robots and connections to social science to engage students and increase participation by under-represented groups.

The goal of this project is to gain a deeper understanding of mobile remote presence systems (MRPs) and to create guidelines for their effective design, development, and adaptation into organizational use. MRP systems enable embodied mediated communication in which individuals at a remote location connect to a local robot that is used to physically navigate in the local environment and to interact with local users via audio and video. MRP systems allow remote users to visit individuals in an organization and attend group meetings, seminars, and social gatherings in the local environment. MRPs enable new forms of interactions and offer remote users an improved sense of presence compared with stationary video-conferencing systems.

The project will study the use of MRPs in communication from an interdisciplinary approach drawing from and building on knowledge and methods from design, social and cognitive psychology, communication studies, and computer science. A series of field and laboratory studies will focus on four topics: (a) how remote users present themselves through MRPs; (b) how local users perceive remote users; (c) the role that social cues play in embodied mediated communication; and (d) the social and organizational outcomes of embodied mediated communication.

Intellectual merit: The project will advance understanding of the role of embodiment in mediated communication in collaborative work and inform the design of future mobile remote presence systems. In addition, the results will contribute to basic science in human-computer and human-robot interaction.

Broader impact: The results will enable more effective embodied mediated communication in organizations, thereby improving collaboration in distributed work groups. The results will also inform the development of tools that help individuals with mobility-related disabilities interact with their social and professional communities. In addition, the project will enhance the undergraduate and graduate curriculum at the University of Wisconsin-Madison, and there will be an outreach program to disperse interdisciplinary knowledge in and methods for designing robotic technology into K-12 education through an annual summer camp and biannual daylong workshops.

Robots hold the promise of delivering significant social contributions in such key domains as education, rehabilitation, and collaboration, but to achieve this promise they must be able to communicate effectively with people. The goal of this project is to design effective social behaviors for robots by modeling human verbal, vocal, and nonverbal behavior using computational tools, regenerating them in robots, and evaluating their effectiveness in human-robot studies in social scenarios such as storytelling, interview, conversation, instruction, and collaborative work. The research will follow an interdisciplinary approach, combining computational and social-scientific methods toward advancing the state of the art in robotic technology.

Intellectual Merit: The research will create a set of design specifications that robot designers can use to create robots that effectively communicate with people. These specifications will be openly disseminated to robot designers and researchers as modules for the open-source Robot Operating System (ROS). The project will also lead to new computational tools that will enable modeling human behavior and synthesizing effective behaviors for robots. In addition, this research will contribute to social science by providing a more detailed, computational understanding of human social behavior.

Broader Impacts: The research will enable robots to communicate more effectively with people in such critical domains as education, collaborative work, and health and wellbeing. The projects will also introduce computational tools into areas that have not yet substantially benefited from advancements in computer technology, such as the diagnosis and treatment of autism and traumatic brain injury. Through an integrated education and outreach program, this project will also contribute to interdisciplinary education, undergraduate and graduate curricula, K-12 and special education, public understanding of science and technology, and further inclusion of minorities and children with disabilities in scientific research and education.

This project explores the hypothesis that compelling learning games based on contemporary science that offer opportunities to contribute to scientific inquiry will lead to increased interest in science, increased career choice of science, increased conceptual understanding of science content, and better scientific literacy around what scientists do. The idea is to capitalize on crowd sourcing both to shed light on the answers to open scientific questions and to engage the public in authentic and needed scientific inquiry in meaningful ways. The PIs will extend four games that are already designed and built or that are under construction and develop a platform for supporting a broad range of participatory science games that offer the public opportunities to contribute to scientific inquiry. The chosen games all encourage sustained and deep participation, include apprenticeship opportunities and opportunities for practicing authentic science, promote reflection in and on action, and are designed to be emotionally compelling. Games come from four game genres: role-playing, strategy, action, and puzzle, as different people are drawn to different types of experiences. All are in the areas of bioscience and biotechnology, and each addresses some open question in bioscience or biotechnology that participants might shed light on. The broad range of games serves several purposes -- offering a substantial enough range of experiences that a broad range of participants can be expected to join in, offering enough diversity to know that the infrastructure tying the games together has all of the functionality required to support a broad range of such games, and offering enough diversity to answer targeted research questions. Research focuses on identifying the challenges in creating a broad and diverse public gaming community that interacts with more formal and established scientific and educational cultures, how learning occurs in such an environment and how to promote sustained engagement and deep learning, identifying core features and mechanisms of games that promote sustained engagement and science learning, and understanding the design features in the particular games being studied that contribute to sustained engagement and learning.

There is an increasing awareness among scientists that many contemporary science problems require (or could benefit tremendously from) an actively engaged public. Communicating the challenges and opportunities of science, and mobilizing the public to participate in and support scientific inquiry, requires shared understandings about the values, methods, and epistemologies of science (e.g., observation, data collection and analysis, reasoning from evidence, skepticism). This project focuses on design of learning opportunities that are both engaging and informative with respect to scientific literacy. The public is invited to participate in a variety of science-related "games," experiences with scientific inquiry that are engaging and exciting and that can contribute to scientific findings. Participants engage as scientists, carrying out the practices of scientists and reasoning about evidence to draw conclusions, in the process experiencing the thrills and frustrations involved in scientific discovery and inquiry. Investigators observe the participants in these games to draw out principles for designing additional learning experiences that can engage the public in science and promote scientific literacy and learning at the same time. What is learned in this analysis will also be applicable to designing engaging science experiences for use in schools.

In order for robots to collaborate efficiently and effectively with humans, the human perception of their movement must be considered in motion creation. Because a human collaborator will interpret the movements of a robot (even subconsciously), robot motion synthesis algorithms that do not consider the human observer may create motions that are perceived incorrectly, interpreted negatively (e.g. as being angry or threatening), or at least miss out on the opportunity to use this subtle communication channel effectively. The key idea of this project is to develop an understanding of human perception of movement that can be applied to the development of robot trajectory planning and control algorithms. The team will use human subjects experiments to understand and evaluate the interpretation of movements and apply these findings in robotics and motion synthesis. The research plan interleaves empirical studies of how people interpret motions, algorithm development to create methods that generate robot motions in a controllable manner, and contextualized deployments that allow the PIs to evaluate the success of the methods. The success of the project will provide a deeper understanding of how people interpret movements, new algorithms for synthesizing robot movements, and demonstrations of the potential applications of collaborative robots.

Broader Impact: Perceptually inspired robot motion synthesis algorithms will enable robots to collaborate more effectively with people. It will enable more communicative robots that can serve as teachers and guides; more approachable and acceptable robots that can work in domestic situations such as elder care; more cooperative robots that can work as assistants to workers; and easier to instruct robots that can be trained by non-experts. This project will enhance the education and outreach efforts of hte PIs by connecting empirical human studies to the technical challenges of robot trajectory planning.

Robots for application in collaborative manufacturing must perform manual work side-by-side with people. Such robots offer the flexibility to work on many different tasks and promise to transform manufacturing by improving the quality and efficiency of manual processes in small shops and in facilitates that manufacture highly customized products. However, in order to meet this promise, robots must be effectively integrated into existing manufacturing teams and practices. To enable this integration, this National Robotics Initiative (NRI) award supports fundamental research on the methods and instruments that manufacturing engineers will need to form effective human-robot teams based on task requirements and worker skills. These methods will also enable robots to adapt to changes in workflow to maximize safety and efficiency. The effective integration of collaborative robots into manufacturing promises improvements in many industries that have not yet benefited from robotic technology. Therefore, results from this research will contribute to the competitiveness of U.S. manufacturing and benefit the U.S. economy and society. The research will involve contributions from multiple disciplines, including robotics, human factors, computer science, and manufacturing, and by academic and industry collaborators. These collaborations will help the dissemination of research results into manufacturing organizations and the integration of research into undergraduate and graduate curriculum in engineering.

Advancements in robotics promise the use of collaborative robots that perform interdependent work with people in order to improve quality, efficiency, and safety in industrial manufacturing. However, integrating collaborative robots into these processes and ensuring their efficient operation pose significant research challenges, including the optimal allocation of work based on task requirements and constraints, the formation of human-robot teams, and the dynamic adaptation of teamwork to workflow changes. This research will address these research challenges, enabling the seamless integration of collaborative robots into these processes and achieving efficient and safe collaboration between human and robot workers. The research team will create novel methods for optimal allocation of tasks to human and robot workers based on task constraints and worker skills, design new tools that utilize these methods to facilitate workflow design for human-robot teams, and develop novel mechanisms that enable robots to more efficiently and safely collaborate with human workers in the planned manufacturing operations. These methods and instruments will be validated in real-world manufacturing operations and disseminated through industry workshops, engineering curricula, and a public outreach program.

Robotics promises to transform how people work, learn, communicate, and engage in other activities of daily living. To do so, robots must interact with users following conventions and norms that people expect, so that people readily adopt them into their environments. These conventions and norms, however, are highly complex and nuanced, requiring that designers consider an impractical number of scenarios, constraints, and requirements in the design process. This project will build new computational methods to facilitate this process, enabling designers of future robot systems to more easily explore the design space for human-robot interactions and to verify that their solutions satisfy design goals. These methods will help achieve the promise of robotics by improving their safety, effectiveness, and reliability.

The design of robotic technologies for human interaction poses a complex, multifaceted design problem that requires that resulting designs satisfy several constraints. Systematically exploring the solution space for actions and behaviors that meet user expectations, follow norms of human interaction, and ensure safety of nearby humans requires an appropriate computational framework for the evaluation and proactive exploration of the space of possible human-robot interactions. This project will devise a novel framework for computationally representing human-robot interactions and exploring the solution space for robot behaviors and actions that satisfy defined correctness properties. Specifically, the project will develop a set of prototypical interaction scenarios and correctness properties; specify formal representations for interaction states and properties, using a variety of temporal logics; apply and extend existing verification methods to verify correctness; devise methods to translate verification output to design feedback; and validate and demonstrate the applicability of the developed framework.

For robots to become ubiquitous collaborators, they must interact physically with objects in unstructured human environments. Robots must adeptly grasp, push, squeeze, snap, balance, stabilize and hand-off objects, and must effectively communicate about these actions with their human collaborators. People must not only be able to specify to a robot what to do and how to do it, they must also be able to interpret what a robot intends to do (and how). Effective communication about physical interactions will be essential if robots are going to perform such tasks as delivering care to older adults, responsively helping technicians in performing repairs, or being trained by non-experts to perform repetitive assembly tasks. This project will enable a new generation of robot applications, and advance the vision of ubiquitous, collaborative robots by developing better methods for robots to more effectively communicate with people.

The project will address key challenges in communication about physical interactions: conveying invisible and unfamiliar quantities (e.g., forces and compliances), communicating plans and contingencies, and communicating about what did not (or should not) happen. The project will involve three key challenges: specification, interpretation, and monitoring. To address these challenges, the project will 1) perform formative studies to gain insight into how people communicate about physical interactions and interpret displays; 2) develop methods for specifying physical actions based on the idea of augmented demonstrations, methods for interpreting physical actions based on the idea of interpretable representations, and methods for monitoring physical actions based on multimodal communication; and 3) deploy these ideas in prototype systems for contextualized scenarios for evaluation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Collaborative robots are emerging as a new family of advanced technologies that are designed to work side-by-side with people in industrial settings. These robots can improve the productivity and flexibility in manufacturing and logistics industries, while also assisting workers in repetitive, and unhealthy or unsafe work conditions. Unlike traditional factory robots that are programmed and configured by engineers to function independently for long periods of time, collaborative robots require human workers to frequently interact with the robot, including training the robot, making the necessary changes in the environment for the robot to function, and supervising the robot's work to ensure its successful operation. The integration of collaborative robots into and operation within existing industrial settings require expert skills that many workers lack, and that current educational programs do not cover, highlighting a skills mismatch from the demands of an increasingly automated future of work. This project will develop an understanding of what skills workers will need to perform these tasks effectively and to design a hybrid digital-physical educational technology that will support worker education and training in these skills in classrooms and industrial-training facilities. The research team will evaluate the effects of the technology on skill development among trainees and make recommendations for future job-training programs.

The new hybrid technology will include an expert system computer program that will provide learners with explanations, visualizations, and simulations of key concepts and a robotic assistant that will provide physical demonstrations. This hybrid physical-digital learning environment will be used in expert-led instruction of traditional and non-traditional students in collaborative robotics. The research team will (1) develop an empirical model of expertise in working with collaborative robots through observations of and interviews with experts and analyses of expert-training programs and (2) iteratively design, build, and test a digital-physical hybrid learning environment, integrating an Expert-View Dashboard (EVD), to support expert-led instruction in educational and industrial-training settings. The resulting educational technology will provide educators and trainers with a powerful educational tool that will supplement the development of expertise in working with autonomous, intelligent, and collaborative technologies to meet the demands of an increasingly automated jobs landscape.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Future educational robots are emerging as social companions supporting learning. By socially interacting with such a robot, learners can potentially reason and talk about the things they are learning and receive help in seeing the relevance of STEM in their daily lives. However, little is known about how to design educational robots to work with youth at home over a long period of time. This project will develop an informal science learning program, called STEMMates, in collaboration with a local community center, for youth with little interest in science. The program will partner learners with an in-home learning companion robot, designed to read books with youth and provide science activities for them at the community center, where youth will engage in exciting and personally relevant science learning. As the learner reads books, the robot will make comments about what is happening in the book to help connect the reading to the science activities at the community center. The overarching goals of STEMMates are to: (a) positively support youth's individual interest in science and future science learning, (b) connect in-home learning experiences with out-of-school community-based learning, (c) bridge the gap between formal and informal engagement and learning in science, and (d) encourage the participation of youth who are underrepresented and who have low interest in STEM learning. This project is funded by the Advancing Informal STEM Learning program, which seeks to advance new approaches to and evidence-based understanding of the design and development of STEM learning opportunities for the public in informal environments.

Researchers will work with youth and staff at the community center, alongside experts in informal science learning, to design the program and then test how learners respond to reading with the robot and participating in the science activities and whether this program has a lasting impact on their science interest. Social interactions with a robot may help distribute cognitive load during learning activities to help youth reason about STEM and also supplement learning by improving feelings of value and belongingness in order to facilitate lasting interest development. Following a mixed-methods research approach using qualitative and quantitative data-collection techniques, the research team will investigate the following research questions: (1) What social and interest-development supports and activities can be utilized as socially situated interest scaffolds in an informal and in-home, augmented reading and science activity program to promote individual interest and learning in science for low interest learners? How can a social robot best facilitate this program? (2) How do learners perceive and interact with the robot in authentic, in-home, long-term situations, and how does this interaction change over time? (3) Does working with a robot designed with socially situated interest scaffolds increase individual interest in science when compared to a pre-intervention baseline, and do these effects impact future (long-term) interest and engagement in formal science learning? To answer these research questions, researchers will implement the science learning program during an 11-week summer deployment and utilize an AB single-case research design. Interview-based qualitative data and self-report surveys to examine the learner?s perception of the robot and their evolving interest in science and quantitative data on science learning using pre-/post-measure comparisons will be collected. Log data of time-on-task, reading rate, book selection and reading goal attainment will also be collected by the robot. The outcomes of this project will lay the groundwork for future investigations of the design of social robots for a diversity of learner populations and their use in different informal learning settings.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The research objective of this Future of Work at the Human-Technology Frontier (FW-HTF) project is to create a matchmaking tool to identify opportunities in manufacturing workplaces where robot capabilities can augment or enhance human workers to improve productivity and safety. This planning project will investigate the possibility to integrate human workers into work processes by modeling tasks from an existing database of critical tasks required to perform each of more than 1000 occupations within the US economy. The project team will also engage with industry to understand differences between the results of task modeling and real work. Successful project outcomes will include the advancement of knowledge around future collaborative technologies, as well as the advancement of fundamental knowledge around the relationship and co-existence between American workers and intelligent robotic workplace assistants. A series of intensive workshops will foster new research collaborations between experts in process optimization, labor economics, and public health to build a multidisciplinary research team for a long-term FW-HTF research initiative.

This planning project will apply state-of-the-art work-design and task-allocation methods and tools to jobs and occupations that show high potential for the integration of collaborative robots to gain a better understanding of the technical challenges in human-robot teaming. The research team will model a subset of candidate tasks from the US Department of Labor Occupational Information Network (O*NET) database and apply optimization-based tools to explore possibilities for human-robot teaming and to investigate the impact of human-robot teaming on both station and system level performance. The project team will analyze the effects of modeled work configurations on key factors such as productivity, ergonomics, and compatibility. Upon project completion, the team will better understand the discrepancy between work models and real-world work, as well as the measurement challenges and constraints in the integration of collaborative robots to existing manufacturing environments. Workshop activities will foster new research collaborations with experts in process optimization, labor economics, and public health to build a multidisciplinary research team for a long-term initiative. The ultimate goal of this project is to develop the necessary research personnel, research infrastructure, and foundational work to expand the opportunities for studying future technology, future workers, and future work at the level of a FW-HTF full research project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This Future of Work at the Human-Technology Frontier (FW-HTF) project advances the vision of robots that work alongside workers to augment human activities, create better working conditions, provide more job satisfaction, open new employment prospects, and result in greater productivity than manual labor or automation alone. Existing principles and methods for work design and automation often use robots to displace human workers and are inherently limited in their economic and social benefits. This research advances a model of robots as physical assistants, working in partnership with human workers to redefine human work. The objective is to create a "matchmaking" process to identify human-robot pairings that can significantly improve productivity, health, and safety, while maximizing human labor resources. A multidisciplinary approach is applied to existing manual tasks in the manufacturing sector that show high potential to benefit from the integration of collaborative robots, in order to gain a better understanding of the impact of teaming humans and intelligent robotic assistants at the individual, operations, and socioeconomic levels. Finally, the insight gained from investigating current job characteristics, worker skills, and robot capabilities, is applied to enhance the benefits of the next generation of manufacturing robots for the strength of the economy and the quality of life of the workforce.

This research aims to provide an increased understanding of how human capabilities can be augmented and enhanced using collaborative robots for physical or cognitive work; a new computational framework to optimally match work characteristics and skills, human capabilities, and robot augmentations; innovative economic models of, and policy recommendations for, the transformation of human employment across industries through human-robot collaboration; and insight into how the next generation of robot systems should be designed. Existing manufacturing sector jobs and tasks that show high potential to benefit from the integration of collaborative robots will be studied to understand how human-robot pairings can significantly improve economic productivity, and worker health and safety. The technical challenges in human-robot teaming are considered in the larger context of worker wellbeing and production as well as the substitutability between labor and robotics, changes in demand for labor skills, and the response in labor demand and supply when technology changes for a specific process. The project includes laboratory simulations and expert review to validate the optimized human-robot work plans, and recommendations on how the next generation of robot systems should be designed to further augment human capabilities and productivity in work.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Researchers in the learning sciences describe learning as a social activity where social interactions with others are conducive to developing deep and lasting knowledge. Teachers often help students using social interactions in science classrooms by providing guidance to students, including giving small cues to what students might notice about their learning, restating key points in what others are explaining, or asking students questions to explain their reasoning. However, during homework, students usually complete lessons by themselves without the aid of rich social interaction that comes from teacher guidance. To help students during homework, educational technologies can provide personalized support for lessons, but most systems lack the ability to provide the type of socially rich guidance that teachers use in classrooms. One solution is to use social robots, or learning companion robots, as an educational technology capable of providing a social learning experience and personalized supports for students as they complete their homework. A major challenge in providing homework lessons designed for social robotic support is a lack of tools for teachers to create and customize these lessons. To address that challenge, in this project, researchers will partner with middle school science teachers and students, to design PATHWiSE, Personalized Augmentation Tools for HomeWork in Science Education, a homework studio for robot augmented learning. The PATHWiSE homework studio is an online system where teachers can design lessons and add robot interactions to science homework and students can log on with a robot in their home to engage in these robot-assisted homework activities. PATHWiSE empowers teachers to add guidance for students during homework by including guiding comments, highlighting material, and asking questions that relate to their classrooms and students lived experiences. With help from the PATHWiSE tool, teachers can transform homework into a socially interactive experience that can vastly improve the learning experience for students of all abilities. This work makes significant societal contributions through an innovative technology that can transform at-home learning experiences; new knowledge and guidance that demonstrate how to develop teacher authoring tools for complex technologies, and new recommendations that inform real-world practice of integrating social robots into science curriculum.<br/><br/>Students usually complete homework lessons in isolation rather than through rich social interaction. Educational technologies can support student homework via step-by-step feedback or proficiency tracking, but most systems lack mechanisms to provide guidance that reflects the socio-cultural experiences students have in and out of the classroom. To address that challenge, researchers in this project will conduct design-based research in partnership with middle school science teachers and students to co-design PATHWiSE, Personalized Augmentation Tools for HomeWork in Science Education, a homework studio for robot augmented learning. The PATHWiSE homework studio is an open-source online system that includes a teacher authoring environment for adding teacher guidance to science homework activities and a student homework environment where students engage in robot-assisted homework activities. PATHWiSE uses a modified trigger-action programming environment that visually overlays trigger-action pairs directly onto multimedia homework activities in the teacher interface. Teachers indicate trigger points, indicating where to provide guidance, and associated actions that describe the robot response at that trigger point. As students work at home with the PATHWiSE interface, the robot will retrieve trigger-action pairs for an assignment from a database. When a trigger point is reached, the associated actions will be translated into robot behaviors (verbal, non-verbal, and movement actions). Finally, researchers will conduct a pilot study to examine teacher use and student experiences with the PATHWiSE system, where teachers will create augmented homework assignments, embedded over four weeks in their actual curriculum, and students will complete these assignments at home with either the robot or a similar computer-based system.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

In modern classrooms, students learn science, technology, engineering, and mathematics (STEM) through interactions not only with teachers and textbooks but also with computer games and simulations, automated tutors, and online resources. Education researchers thus have access to large amounts of data about students’ STEM learning processes, from classroom or online conversations to detailed records of student activity in educational apps. Despite the potential of such rich data for curriculum development and personalized assessment, there are significant technical and conceptual challenges to analyzing data that come from different sources or modalities. To address these challenges, this project will develop and test trans-modal analysis (TMA). TMA is a statistical technique and software package that will help researchers better and more easily integrate multiple types of data into analyses of STEM learning. This will enable more accurate understanding of students’ STEM learning processes, and in turn help identify potential inequities in assessment of student learning, informing education policy and practice for diverse learners. This project will include post-doctoral scholars, graduate student researchers, and undergraduate research interns, who will develop skills and experience in data science, learning analytics, software development, and scientific communication, providing training and mentoring for the next generation of education researchers. <br/><br/>Although most analyses of learning processes are based on a single type or modality of data, STEM learning typically takes place in a multimodal setting. Models of STEM learning processes thus need to account for multiple sources and types of data to account for complex interactions between learners and the setting(s) in which they learn. For example, there are different types of events (questions from a teacher, chats with a peer, views of a resource) and different properties of events (gender of a person gesturing, linguistic fluency of a speaker, reading level of a person reading a document) that may influence future events with more or less impact over time. In addition, the structure of a learning environment creates a horizon of observation for each student, making some events (e.g., a conversation in another group of students) more or less visible. Finally, different characteristics of students (age, cultural or ethnic background, gender identification, whether instruction is in their native language or a non-native language) may lead them to respond to events in different ways. Extant learning analytic techniques account for the influence of prior events by lagging: for example, using some fixed number of prior events to predict future events. TMA will enable those same techniques to operate not on properties of the events themselves but on underlying functions that represent claims or hypotheses about the interaction between different learning modalities, the structure of the learning environment, and the ways in which students might systematically differ as STEM learners. The project team hypothesizes that TMA models will provide a more nuanced, more accurate, and more equitable view of STEM learning processes for diverse learners. This approach will expand the understanding of effective multi-modal STEM learning processes and allow researchers to account for diversity and address questions of equity in multi-modal STEM learning. TMA will be developed and tested first as a set of algorithms for conducting trans-modal analyses with three widely used learning analytic tools: process mining, epistemic network analysis, and dynamic Bayesian networks. The investigators aim to use simulation studies and the analysis of actual STEM learning datasets to address two fundamental research questions regarding the science of learning: (1) Under what conditions (if any) are trans-modal models of STEM learning processes more informative than uni-modal models? And (2) Can TMA model meaningful differences in trans-modal learning processes for minoritized groups of STEM learners?<br/><br/>This project is supported by NSF's EHR Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development. The program supports the accumulation of robust evidence to inform efforts to understand, build theory to explain, and suggest intervention and innovations to address persistent challenges in education.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.