Darren R Gergle, PhD - US grants

With the advent of increasing numbers of increasingly smart machines, there is a growing need to develop technologies that are not only smart, but sensitive to the people and the other machines around them, and sensitive to the context in which they are used. Such an understanding will permit the development of technologies that can coordinate their interactions with humans in a more natural, seamless and fluid fashion. To meet these goals, this research program focuses on three critical yet under-studied contexts of interaction, each of which represents a different constraint upon interpersonal communication: (1) the physical context of shared visual access, (2) the social context of rapport, and (3) the biological context of aging. While some research has been conducted on each of these contextual factors, none has addressed their interaction, nor gathered them into one broader conception of the role of context in interpersonal coordination. This research applies a theory-driven design approach that includes experimental studies, theory development, computational modeling, system implementation and evaluation. In particular, the research program proposes: a) A rigorous study of human-to-human communication using elicitation experiments to develop a more detailed understanding of interpersonal communication across a range of contexts; b) A formalization of the findings into computationally explicit forms that provide predictions of behavior and capture the observed behavioral patterns; c) Integration of the models into a dialogue manager that is implemented within a larger computational architecture; and, d) Evaluation of the implemented system by having untrained humans interact with the system in such a way as to evaluate its effectiveness and reveal gaps in the underlying models as well as in our theoretical understanding.

The outcome of this research will advance our theoretical understanding of the role various contextual factors play during interpersonal communication. The results will be useful to a variety of scientific communities including those that study basic human communication (e.g., psychologists, linguists and communication researchers) and those that study interactive computational systems (e.g., computer scientists, computational linguists, and interaction designers). The research will also provide practical design guidelines and a general computational model that describes how machines can make intelligent choices on the basis of these contextual factors during everyday interactions. At a practical level, the general computational model can be applied by technologists developing many different technologies, such as embodied agents, large-scale displays, ubiquitous computing, in-car navigation, and assistive technologies for the elderly and those with cognitive impairments.

This research aims to enable the development of the next generation of collaborative technologies and to study their effects on human collaboration. It does so by improving our theoretical understanding of how various features of shared visual context affect communication, coordination and collaboration. The research develops and makes available the code for a new dual eye tracking methodology that can be used to study coordination and collaboration. It also develops a new set of metrics that can be more generally used to understand coordinated eye movements as they relate to dialogue and conversation. The results of this work will add to knowledge in a number of disciplines, including human-computer interaction, language technologies, computer science, computational linguistics, communication studies, cognitive science, and social and cognitive psychology.

As recent efforts in telemedicine, distance education, and remote training and repair attest, there exists enormous potential for technologies to support these activities at a distance, ultimately resulting in widespread benefits such as equal access to quality education and medical care. The education activities in this research include: (1) developing a new Ph.D. course that teaches a theory-driven design approach; (2) developing a series of course modules that use the dual eye tracking methodology to teach behavioral coding, statistics and machine learning; (3) developing a publicly available multimodal corpus with a set of tutorials, and (4) developing a series of international workshops on dual eye tracking. Finally, the research will simultaneously advance discovery and understanding while training both graduate and undergraduate students in interdisciplinary research methods and will contribute to the development of traditionally underrepresented individuals in the fields of computer and information sciences.

The United States is a world-leader in software and in multimedia content (e.g. music, film). To remain so, we must continually raise the bar in both software and media production. Software tools for media production (e.g. the audio production suite Protools) often have complex interfaces, conceptualized in ways that makes it difficult for any but the most expert to realize the power of these tools. Complex interfaces and steep learning curves can discourage creative people from doing their best work with such tools. Here, we focus on audio production tools. We propose a user-centered approach to remove the great disconnect between existing audio production tools and the conceptual frameworks within which many people work, both expert musicians and the broader public. The tools we develop will automatically adapt to the user's conceptual framework, rather than forcing the user to adapt to the tools. Where appropriate, the tools will speed and enhance their adaptation using active learning informed by interaction with previous users (transfer learning). The tools will also automatically build a crowdsourced audio concept map. This will help provide facilities for computer-aided, directed learning, so that tool users can expand their conceptual frameworks and abilities. By letting people manipulate audio on their own terms and enhancing their knowledge of such tools with directed learning, we expect to transform the interaction experience, making the computer a device that supports and enhances creativity, rather than an obstacle.

This work will have a number of broader impacts. The tools developed will be directly usable by practicing musicians and will also facilitate learning and creativity for the general public. These techniques will also be applicable to personalization of hearing aids and new diagnostic systems for audiologists. Our approach to tool personalization is core work in human-computer interaction and should generalize to other creative activities (e.g. image manipulation). Resulting advances in active and transfer learning will be of great value to machine learning researchers. Finding the relationships between quantifiable parameters of audio and the language and metaphors used by practicing musicians to describe sound is central to this work. This is of great interest to cognitive scientists, linguists, artificial intelligence researchers, and engineers. Concept maps for audio terms should also prove useful for machine translation. Broad application of techniques to map human descriptive terms on to machine-manipulable parameters will change expectations for both artists and scientists. Artists will be able to explore new lines of creativity that currently require significant investments of time in vastly disparate fields (e.g. signal processing and painting). This has the potential to transform information science and lead to new cognitive models of creativity, forming the basis for new approaches to education and research in both technology and in art.

This project will develop theory, algorithms, and example systems for motivating large numbers of autonomous individuals to solve problems that require coordination in the physical world. In these "physical crowdsourcing" systems, people make small contributions toward a larger collective problem, such as tracking animal species or air quality for citizen science projects or providing rides or package delivery in commercial applications. In these systems, opportunistically relying on people to do convenient parts of the problem causes incomplete solutions, while directing people to do inconvenient tasks requires high incentives. By modeling the timing and location of tasks, along with knowledge of people's routines, the project will develop algorithms that make use of ideas from decision theory to best decide when, where and to whom to suggest tasks, in a way that balances individual convenience with system needs. Doing this should increase people's willingness to participate, reduce the need to incentivize participation, and create more complete, timely, and accurate solutions to the collective problem. The investigators will develop several prototype systems that address practical problems such as package delivery and lost and found searches to demonstrate the effectiveness of their ideas; they will also release a toolkit that allows other people to use their work when designing their own physical crowdsourcing systems for solving both scientific and practical problems of interest to society.

The work will be carried out in three main phases. The first phase will develop "incentive chaining", a strategy to encourage opportunistic contributors to become more directable. Using an existing prototype for surveying tree species, the team will model people's opportunistic contributions and routines, then suggest relatively easy "nearby" tasks where others have contributed but where further work is needed, building people's interest and capacity to contribute. Once the nearby collaborative tasks are done, the system will direct these experienced and motivated contributors toward new areas and tasks, which will in turn become the "nearby" contributions that attract the next round of contributors. The second phase will develop decision-theoretic hit-or-wait algorithms to decide if and when to notify contributors about tasks, using a community-based lost and found application. Building on the modeling of behavioral routines, these algorithms will use sequential decision processes to estimate the maximize total value of tasks that contributors will complete, given individuals' history and current state, the system-determined value of nearby tasks, and the system's estimate of how likely the contributor would be to respond to a notification. The third phase will develop notification policies that satisfy the system's quality of service needs while minimizing disruption to contributors. To do this, the investigators will develop a supply management framework that considers the pool of available helpers, the current task demand and values, and a range of policies for making directed requests and evaluate it in the context of a peer-to-peer package delivery system.