Jodi Forlizzi - US grants

Soon, autonomous mobile robots will be used in settings such as elderly communities, schools, and hospitals to carry out intelligent but "unskilled" tasks such as delivering food or teaching calisthenics. Humans are integral in these systems-as operators, as users, and as people who live and work where robots are employed. This proposal targets critical questions of human-robot interaction, and of robotic assistants in personal and work settings. The research involves experiments on mental models of robots and robotic assistants, ethnographic design research, and organizational field research. The initial research will aim at design of appropriate roles, tasks, and interactions of robotic assistants in elderly environments. Later research will address other domains for using assistive robots. This research builds on cognitive and social psychology, and design. The research will contribute to theory on people's interactions with robots, facilitate useful and graceful interactions between people and robotic assistants, and advance robotic technology and dialogue on ethical issues surrounding deployment of life-like robots. The research team is highly multidisciplinary. This project will address major NSF concerns including: bringing technology to bear for special populations, extending human capabilities, using technology for collective work, and integrating technology into social contexts.

We are poised at the threshold of an information rich world with devices and services able to deliver that information to nearly anyone, at any place, and at any time. Humans have evolved social mechanisms for smoothly and flexibly managing interpersonal communications; however, current computational and communications devices are, almost without exception, utterly unaware of the social and attentional state of the user. They know little or nothing of the personal, social, and task situations in which they are used, and they do little or nothing to account for, and minimize, the human costs they induce. In this project, the PI and his team will explore situationally appropriate interfaces that retrieve, generate, and deliver information in a manner that is sensitive to the situation of the user. These interfaces will allow for communication and information systems that maneuver, rather than blunder, through the social world. To accomplish this ambitious goal, the team will pursue a three-part research plan. First, they will use behavioral theory and research to model social mechanisms for managing interpersonal communications. The comparatively unexploited research we will draw on examines the affordances of situations and consistent patterns of human nonverbal social behavior within situations. Second, they will extract key situational and user behavior data from these models via input from new sensing technologies, using noninvasive (e.g., vision-based) sensing technology to provide information about situations and users. Third, leveraging knowledge from sensory, perceptual, and cognitive psychology, as well as from the fields of visual and interaction design, the team will create displays and interaction designs that are far more situationally appropriate than today's interfaces. To address the substantial challenges that this breadth of work presents, the PI has assembled a strong multidisciplinary team that brings expertise from computer science, social, sensory, perceptual, and cognitive psychology, and the field of design.

Since at least the early 1900s, the supply of information and communication has been growing faster than our ability to consume it. From the information producers' viewpoint, much information is wasted, in the sense that it doesn't reach those to whom it is relevant. For the consumer, the glut of information makes it increasingly difficult to find what is relevant, useful or enjoyable. This "poverty of attention" also occurs in the domain of interpersonal communication, where informal and spontaneous interaction, a hallmark of managerial and professional work, generates interruptions and overload. In this project, the PI and his team will develop, integrate and use principles from social psychology, computer science, economics, and interaction design to devise ways of economizing on attention in a communication rich environment. Specifically, they will develop, deploy and evaluate techniques to mediate among the often competing demands of responding to a barrage of communication requests that compete for attention when performing a primary task, such as driving a car or interviewing a client. To do so, the PI team will use sensing technology and machine learning techniques to identify a worker's current task and degree of interruptibility. They will display team members' availability to a subset of communication partners and use principles developed in economics and social psychology to balance the competing demands on the worker's attention. They will also use principles developed in interaction design to build lightweight, peripheral displays and context-aware interaction techniques that allow people to briefly and simply comprehend information without disrupting their primary task. Although the principles and techniques developed will be applicable across many domains, this project will make use of a specific context for development and testing, namely improving communication efficiency of mobile health care workers caring for the elderly. The PI team consists of a social psychologist, a computer scientist, and a designer, who have been working previously both individually and jointly on problems of human attention; part of this project will involve integrating results from research efforts already underway.

Broader Impacts: Issues of managing attention and information overload are pervasive in our increasingly connected and technological society. Development of principles and specific techniques for addressing these issues has the potential for large benefit and widespread impact beyond the eldercare domain. In addition, support for the project's target domain - mobile eldercare workers - offers the potential of direct social benefit to a rapidly aging population. The project will lead to better understanding of design principles for economizing on attention, to prototype development, and to lab and field testing of the models, all of which will help save on the most precious of human resources - our attention.

This is a project to understand and test theories of how three aspects of human-robot interaction independently and interactively affect collaborative work: robot social behavior, mutual understanding in human-robot communication, and the impact of robotic assistants on group dynamics. With advances in computing technology and artificial intelligence, autonomous robots are becoming viable in such critical domains as search and rescue, military battle, mine and bomb detection, scientific exploration, law enforcement, and hospital care. Robotic assistants ranging from museum guides to forestry scouts are being developed to interact with people "in person" or remotely, as agents that collaborate with the work team. This project is targeted at little understood but critical aspects of robot-supported collaborative work.

The research involves three kinds of studies: (1) fundamental laboratory research on behavioral characteristics of robots performing social tasks, especially as these characteristics reflect lifelikeness, (2) controlled experiments and field studies of interpersonal communication and the development of mutual understanding between robot and human, and (3) studies of robots in work groups. The studies are designed to motivate and test theory, as well as to explore both direct and secondary or indirect social effects of robot-supported collaborative work. The fieldwork will be carried out in hospitals and scientific exploration settings.

This research will advance our understanding of the possibilities and problems of mutual adaptation in human-robot interaction over time, and will help us anticipate changes in the group dynamics of collaborative work. It also will extend our basic knowledge of communication and group dynamics in environments that incorporate robotic technology. It will improve our ability to make principled design decisions about robots that work with people, and better understand the societal impact of robots. In particular, this research will give us a foundation for understanding and designing collaborative work with robots in critical environments like mines, hospitals, households with elderly or disabled residents, in challenging scientific settings, and in situations in which the robot is remote. The work also will contribute to public and student awareness of the human side of robotics, and help motivate students' interests in science and engineering.

This project enhances the expressive ability of the fundamental medium of text with the development of tools that enhance the creativity of people while generating kinetic typography. Traditional forms of static typography can be augmented with time and motion, in order to manipulate the position, size, color, shape, and other properties of text displays over time. This new kinetic typography offers a number of potential advantages: the ability to convey emotional content and qualities of the speaker's voice, the potential for increased reading performance on very small displays, and the ability to communicate in new ways. These advantages have been largely unexploited because the technological tools needed to make kinetic typography easily accessible to both designers and the general public have only begun to be developed. This project explores the nature of tools that enhance human creativity in the context of generating kinetic text. The impact of this work will reach well beyond kinetic typography, with implications for understanding the design process, emotional communication, and creativity. This research will extend our basic knowledge of communication and information transfer, and give us a foundation for understanding and creating kinetic typography tools, with long-term implications for work in design schools and training environments.

Abstract

This project at Carnegie Mellon University tests an educational game aimed at addressing problems students face learning mathematics. The PI plans to design, develop, iteratively test, and scientifically evaluate a potentially transformative gaming technology. This approach combines an educational game environment, erroneous mathematics examples, state-of-the-art interaction design, and adaptive tutoring technology to help engage and motivate students to learn mathematics. The project develops a blueprint for educational games that can be used with middle school students in both formal and informal settings. Two research questions guide the research: Can erroneous examples increase learning and add motivation to an educational game? Will making erroneous examples adaptive to individual students increase learning and motivation beyond the non-adaptive version of the educational game?

The project randomly assigns 6th, 7th, and 8th grade students to an intervention that brings together design thinking and design activity in co-designing sessions involving students and their teachers. This approach allows students and teachers to create prototypes of specific interfaces for mathematical games using results from the co-designing sessions. The prototypes involves developing personas, scenarios, wire-framing, rapid prototyping, and speed dating. It also includes rapid concept evaluation sessions created iteratively with students and their teachers to inform the final game design. An online software instrument logs student data from pre-questionnaire, pretest, game playing, posttest, delayed post-test, and post-questionnaire. The software also logs correct and incorrect attempts at solving problems, evaluations of these attempts by the tutor, the number of hints requested, and the amount of time spent on task. These data provide immediate feedback to teachers and researchers about student understanding of specific concepts in mathematics. The project assembles a multi-disciplinary team of experts from learning domains using erroneous examples, educational technology, intelligent tutoring systems, human-computer interaction and interaction design, prolific experimental educational psychologist, and a cognitive scientist with a long history and knowledge of mathematics education. The expectation is that these partners will increase the effectiveness of the design outcome as proven in other co-design studies.

The workshop will consider future innovations in transportation technology and policies in the Arctic that could address challenges associated with rapid climate change. One transportation challenge, for example, is the limited network of roads which are now being undermined by permafrost thaw and flooding; while another challenge is the subsistence based rural economies which may face food shortages. The workshop will bring together a number of natural and social scientists and engineers with relevant expertise to consider how autonomous and robotic transportation might address transportation challenges of the new Arctic in the context of social, cultural, economic and environmental systems.

The workshop will consider future innovations in transportation technology and policies in the Arctic that could address challenges associated with rapid climate change. This workshop proposes not only to look at the technological and engineering advancements of robotic transportation but to expand collaborations between engineers, social and natural scientists in order to develop new convergent ways of thinking about this technology and its social and environmental impacts. These issues are particularly relevant in the north because of the long distances, rural economies, fragile environments, and majority Indigenous communities that pose both opportunities and challenges to robotic transportation, particularly within the context of globalizing social and economic systems and environmental change. The workshop will be structured to enhance convergent outcomes through a combination of lightning talks, co-design exercises and recombining the initial organizing committees into convergence teams. The workshop will promote convergence by focusing on the critical societal concern of limitations to inland transportation in the Arctic and by bringing together experts from many relevant disciplines of social science, natural science and engineering to devise new creative approaches to address these limitations.

This Early-concept Grant for Exploratory Research aims to help patients and caregivers have increased access to electronic health information. The research focuses on the Electronic Health Record (EHR). In 1996, the Health Insurance Portability and Accountability Act entitled patients the right to obtain their medical records on request. With greater patient access, patients may become more invested in the management of their own health care and take better care of themselves. While there have been some advances in communicating EHR content to patients, physicians' and nurses' notes are not easy to convey due to communication intent of clinicians and the social and cultural issues that surround diagnosis and care. Advances from this work will reduce gaps in the care process, potentially impact consumer side patient care, and become the basis for future systems development.

The research investigates new and more effective presentations of this information to patients (e.g., graphic, abstracted, actionable). The project includes the following activities: 1) collect patient notes and the communicative intent for both clinicians and patients to reveal the gap between intention and interpretation; 2) iterate on new textual and visual summaries by hand until they achieve the intended communication with patients. The proposal team's expertise will be leveraged in designing abstractions of information to help people make decisions and positively change their behavior under stress or uncertainty to undertake these activities.

Design innovations follow technical advances. Philips? 1962 invention of the cassette recorder was a technical advance. Design innovations followed, placing cassette audio recording and playback capabilities into new forms people found useful and valuable. These included the boombox, car stereo, Walkman, and home answering machine. Artificial intelligence (AI) has achieved many technical advances, but insufficient design innovation. To enable their innovation capabilities, design teams need to better understanding what AI can do. This research will address current breakdowns in the innovation process. It will develop new taxonomies, tools, and theory?to unleash the creative skills of designers in industry and academia?for envisioning and prototyping innovative uses and forms for AI products and services. The research team will disseminate refined versions of the new AI design innovation resources through online training modules, undergraduate and graduate courses, courses for professional user experience (UX) designers, and workshops with professionals at practitioner-focused events.

For AI innovation, the research team will perform these design activities: (1) develop and evaluate a taxonomy of AI capabilities, while addressing algorithmic bias and privacy; (2) develop and evaluate a taxonomy of design patterns for how AI system outputs are presented to potential innovators, including approaches to explainable AI; (3) co-design new tools for wireframing AI system behaviors, documenting AI concepts, and communicating with and across AI development teams, through prototyping workshops with professional UX designers; and (4) develop theory and design implications through field evaluations of the AI design innovation taxonomies, tools, and communicative forms.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This Future of Work at the Human-Technology frontier planning grant focuses on creating new ways for hospitality workers to shape how automation will affect their industry and their jobs. It has been widely predicted that the hospitality industry will experience significant job shifts in the coming decades. Automation plays a key role in causing these shifts, with examples including self-service kiosks, various applications of robotics, customer-facing AI, automated operations, and food preparation technologies. Such automation is already displacing many jobs in the hospitality industry. Although new jobs will be developed in the future, many others will be eliminated, and the need for new skill sets is rising. In the hospitality and service industries, automation has already affected hotel room check-in, cashiering, cleaning processes, bartending, and food service. This situation is changing even more rapidly as the work shifts in response to new standards around COVID-19 safety. Nearly 90% of hospitality workers lost work at least temporarily during the pandemic and they lack ways to envision and reap the positive impacts of technology innovation.

The research team of experts in robotics, policy, economics, human-computer interaction, and organizational dynamics seeks to mitigate this growing problem by allowing hospitality workers to partner with and benefit from the future of automation and to maximize opportunities and positive outcomes for all of its stakeholders. The project will create generalizable knowledge through a series of workshops that bring together researchers, policy experts, union leaders and members, and stakeholders from leading hospitality schools. Specifically, the research team will assess the current state of hospitality services through a broad literature review, conduct workshops with key stakeholders, collectively explore technology innovations, and develop a strategic roadmap for future collaboration around automation technologies in the hospitality industry. This project has been funded by the NSF Future of Work at the Human-Technology Frontier cross-directorate program to promote deeper basic understanding of the interdependent human-technology partnership in work contexts by advancing design of intelligent work technologies that operate in harmony with human workers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project centers consideration of privacy in the development of artificial intelligence (AI) technologies that directly impact consumers. Using AI can help systems adapt to specific people's goals and abilities; however, AI systems typically require collecting data and making guesses about their users, both of which can be intrusive and cause harms. For instance, AI systems sometimes make wrong inferences about personal, sensitive characteristics that can cause both psychological harm and affect people's access to systems; the data collected can also be used in unwanted ways, such as large facial recognition databases assembled without people's consent. These harms often happen, even when system designers are well-intentioned, because current design practice provides little specific guidance on how to reason about possible harms. This project will tackle this problem by creating design methods and guidelines that highlight potential privacy issues and design choices that often increase these risks. Student and industry researcher involvement in the development and evaluation of the methods will give the work both direct educational impact and increase the chance that future AI-based systems will make informed choices around privacy and safety risks.

The specific method proposed is called Privacy through Design (PtD), a novel research methodology to help creators of consumer-facing AI technologies: (i) model how acute, use-case specific privacy concerns among end-users among stakeholders trade off against the envisioned utility or value of proposed AI concepts; and, (ii) understand how to (re-)design those concepts in a manner that respects stakeholders' privacy concerns of while retaining the envisioned utility of the design. Doing this work makes three main scientific contributions. The first is to develop a taxonomy of algorithmic privacy intrusions to operationalize the unique privacy harms entailed by consumer AI and map those harms onto the unique capabilities and requirements of AI systems. This second is to develop PtD using an iterative methodology incorporating experts and practitioners in industry and academia. The third is to formally evaluate how products developed through PtD compare to those developed through existing industry standards for designing consumer AI technologies. Two key envisioned outputs are a repository of design cases in which privacy concerns emerge and are resolved, and a guidebook with worksheets and recommendations to help creators of consumer AI technologies center consideration of privacy in their design processes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This NSF S&CC planning grant contributes to the design and development of new mobility technologies that serve city residents. New mobility technologies are on the rise in cities across the US, with micromobility platforms and autonomous delivery robots occupying the sidewalks of many cities. Amid the enthusiasm around what these devices could mean for the future of transportation and the circulation of essential goods, are concerns about who stands to benefit from these technologies. This research focuses on producing a robust understanding of the unintended harms these devices may introduce and, in collaboration with community members, offer a set of potential opportunities to support resident safety and ensure broad community benefit

Pennsylvania recently passed state-wide legislation classifying personal delivery devices (PDDs) as "pedestrians," bestowing them the same legal rights as human residents. In response, Pittsburgh government officials and residents have come together as a coalition with the aim of developing municipal standards on current delivery devices to ensure continued accessibility of public space, and strategies for planning proactively for future new mobility deployments in the area. Researchers in this project have partnered with the City of Pittsburgh’s Department of Mobility and Infrastructure (DOMI) and established connections with local advocacy organizations to develop community-driven approaches to the deployment and governance of new mobility technologies. Extending methods of impact analysis and equity measures, they draw insights from the actions and perspectives of government officials and residents who are negotiating the changing shape of the city and seeking to define community-driven technology deployments. This planning grant lays the groundwork for a longer-term research with their Pittsburgh partners and other cities, by developing methods and technologies for supporting democratic processes around public PDD deployments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This multi-institutional collaborative research project seeks to envision, design and engage in worker-oriented research and training related to the proliferation of automation in the hospitality industry, which has been accelerated by the COVID-19 pandemic. Large numbers of hospitality workers, who are majority female and from underrepresented groups, are being displaced by these technological changes. Workers’ positions are currently being augmented with algorithmic management and robotic assistance, replacing some jobs and transforming others that cannot be completely automated — for example, high-touch, face-to-face service interactions which are critical to the success of the hospitality industry. The research team will investigate ways that technological innovations of the future can be developed and implemented with input from the workers who are best suited to understand their benefits and pitfalls. It is expected that this project will positively impact the hospitality workforce by preserving jobs, giving people more job satisfaction, reducing the way technology drives inequality, and developing policies and training programs that will generalize to other high-touch service industries.

This project brings together several disciplines, including hospitality, human-computer interaction, service design, learning science, labor economics, and industrial relations. The research team has partnered with UNITE HERE, the largest hospitality union in the United States, which will provide a unique opportunity to research, prototype, and evaluate the research outcomes in training facilities and hotels, casinos, and food service establishments. The investigator team is structured to achieve multiple convergent goals. This process will have four iterative, overlapping phases: (1) to qualitatively and quantitatively understand the current state of union hospitality workers, hospitality work, and automation technology; (2) to co-design technology deployment models; (3) to identify workforce needs and training materials to prepare for the future; and (4) to evaluate outcomes to understand how they may impact the future of work. The project will address the lack of worker voice on the impact of future automation technology in the hospitality industry, and to make suggestions for enhancing future workers and future work. This project is funded by the NSF Future of Work at the Human-Technology Frontier cross-directorate program to promote deeper basic understanding of the interdependent human-technology partnership in work contexts by advancing the design of intelligent work technologies that operate in harmony with human workers.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.