Darren R Gergle, PhD - US grants

With the advent of increasing numbers of increasingly smart machines, there is a growing need to develop technologies that are not only smart, but sensitive to the people and the other machines around them, and sensitive to the context in which they are used. Such an understanding will permit the development of technologies that can coordinate their interactions with humans in a more natural, seamless and fluid fashion. To meet these goals, this research program focuses on three critical yet under-studied contexts of interaction, each of which represents a different constraint upon interpersonal communication: (1) the physical context of shared visual access, (2) the social context of rapport, and (3) the biological context of aging. While some research has been conducted on each of these contextual factors, none has addressed their interaction, nor gathered them into one broader conception of the role of context in interpersonal coordination. This research applies a theory-driven design approach that includes experimental studies, theory development, computational modeling, system implementation and evaluation. In particular, the research program proposes: a) A rigorous study of human-to-human communication using elicitation experiments to develop a more detailed understanding of interpersonal communication across a range of contexts; b) A formalization of the findings into computationally explicit forms that provide predictions of behavior and capture the observed behavioral patterns; c) Integration of the models into a dialogue manager that is implemented within a larger computational architecture; and, d) Evaluation of the implemented system by having untrained humans interact with the system in such a way as to evaluate its effectiveness and reveal gaps in the underlying models as well as in our theoretical understanding.

The outcome of this research will advance our theoretical understanding of the role various contextual factors play during interpersonal communication. The results will be useful to a variety of scientific communities including those that study basic human communication (e.g., psychologists, linguists and communication researchers) and those that study interactive computational systems (e.g., computer scientists, computational linguists, and interaction designers). The research will also provide practical design guidelines and a general computational model that describes how machines can make intelligent choices on the basis of these contextual factors during everyday interactions. At a practical level, the general computational model can be applied by technologists developing many different technologies, such as embodied agents, large-scale displays, ubiquitous computing, in-car navigation, and assistive technologies for the elderly and those with cognitive impairments.

The growing power and flexibility of information technology has allowed developers to build increasingly creative tools; however that same capacity has often led to tools whose use is beyond any but the most expert power-user. One area of creative endeavor of great cultural and economic importance is music. In recent decades, a vast variety of audio production tools have been introduced to enhance and facilitate music creation. Unfortunately, since the tools come from software developers more familiar with Java than musical creativity, many of these tools are extremely complex and use concepts and controllers very different from those that musicians are accustomed to and comfortable with. Since the tools are complex, and interfaces to software suites change every few years, musicians may fail to embrace them or only ?half- learn? them, inhibiting creativity and limiting the range of possible works produced, and the ultimate capacity that powerful technology has to enhance musical creation. Many musicians think about sound in individualistic terms that may not have known mappings onto the controls of existing audio production tools. For example, a violinist may want to make the recording of her violin sound shimmery. While she has a clear concept of what a shimmery sound is, she may not know how to articulate it in terms that let a producer map shimmery onto the available audio tools (such as reverberation and equalization). This project develops a computational tool that works alongside the musician to quickly learn how acoustic features map onto an audio concept, and creates a simple controller to manipulate audio in terms of that concept. In the case of the violin player, the tool would learn what shimmery means to her, and then create a knob that would let her make a sound more or less shimmery. This project uses a user-centered design approach to the development of audio production tools that automatically adapts to the user?s work style, rather than forcing the user to adapt to the tools. The result will be and example of how audio tools can enhance creativity, and more broadly, how affective concepts can be automatically mapped to more discrete parameters associated with digital media production.

This research aims to enable the development of the next generation of collaborative technologies and to study their effects on human collaboration. It does so by improving our theoretical understanding of how various features of shared visual context affect communication, coordination and collaboration. The research develops and makes available the code for a new dual eye tracking methodology that can be used to study coordination and collaboration. It also develops a new set of metrics that can be more generally used to understand coordinated eye movements as they relate to dialogue and conversation. The results of this work will add to knowledge in a number of disciplines, including human-computer interaction, language technologies, computer science, computational linguistics, communication studies, cognitive science, and social and cognitive psychology.

As recent efforts in telemedicine, distance education, and remote training and repair attest, there exists enormous potential for technologies to support these activities at a distance, ultimately resulting in widespread benefits such as equal access to quality education and medical care. The education activities in this research include: (1) developing a new Ph.D. course that teaches a theory-driven design approach; (2) developing a series of course modules that use the dual eye tracking methodology to teach behavioral coding, statistics and machine learning; (3) developing a publicly available multimodal corpus with a set of tutorials, and (4) developing a series of international workshops on dual eye tracking. Finally, the research will simultaneously advance discovery and understanding while training both graduate and undergraduate students in interdisciplinary research methods and will contribute to the development of traditionally underrepresented individuals in the fields of computer and information sciences.

Developed around the research theme of computer innovation in modeling and simulation of complex human and physical systems, co-PIs and their students will work with local classroom teachers with the goal of taking cutting-edge modeling and simulation from the fields of computer science, physics and astronomy and translate them into tools that can teach grade-level appropriate science content. Each graduate fellow will be partnered with a local classroom teacher and will engage in a training, development, and implementation process over the entire year. External evaluators will help us assess progress towards the goals and benchmarks of the program.

More broadly, this work will build the graduate fellow?s teaching, analytical and communication skills. The fellows will gain a deeper understanding of the foundations of their research in deconstructing and isolating the basic concepts of computer science, physics, and astronomy from their work to teach them to the classroom students. Classroom teachers will be exposed to cutting-edge scientific research and the research process as well as forming lasting partnerships with university researchers and scientists. Classroom students, many from traditionally underrepresented populations, will gain an improved understanding of science content and the research process, enabling them to be more informed consumers of science content presented in the media. Classroom students will also be encouraged to pursue careers in science, technology, mathematics and engineering from interactions with the fellows as role models and mentors.

The United States is a world-leader in software and in multimedia content (e.g. music, film). To remain so, we must continually raise the bar in both software and media production. Software tools for media production (e.g. the audio production suite Protools) often have complex interfaces, conceptualized in ways that makes it difficult for any but the most expert to realize the power of these tools. Complex interfaces and steep learning curves can discourage creative people from doing their best work with such tools. Here, we focus on audio production tools. We propose a user-centered approach to remove the great disconnect between existing audio production tools and the conceptual frameworks within which many people work, both expert musicians and the broader public. The tools we develop will automatically adapt to the user's conceptual framework, rather than forcing the user to adapt to the tools. Where appropriate, the tools will speed and enhance their adaptation using active learning informed by interaction with previous users (transfer learning). The tools will also automatically build a crowdsourced audio concept map. This will help provide facilities for computer-aided, directed learning, so that tool users can expand their conceptual frameworks and abilities. By letting people manipulate audio on their own terms and enhancing their knowledge of such tools with directed learning, we expect to transform the interaction experience, making the computer a device that supports and enhances creativity, rather than an obstacle.

This work will have a number of broader impacts. The tools developed will be directly usable by practicing musicians and will also facilitate learning and creativity for the general public. These techniques will also be applicable to personalization of hearing aids and new diagnostic systems for audiologists. Our approach to tool personalization is core work in human-computer interaction and should generalize to other creative activities (e.g. image manipulation). Resulting advances in active and transfer learning will be of great value to machine learning researchers. Finding the relationships between quantifiable parameters of audio and the language and metaphors used by practicing musicians to describe sound is central to this work. This is of great interest to cognitive scientists, linguists, artificial intelligence researchers, and engineers. Concept maps for audio terms should also prove useful for machine translation. Broad application of techniques to map human descriptive terms on to machine-manipulable parameters will change expectations for both artists and scientists. Artists will be able to explore new lines of creativity that currently require significant investments of time in vastly disparate fields (e.g. signal processing and painting). This has the potential to transform information science and lead to new cognitive models of creativity, forming the basis for new approaches to education and research in both technology and in art.

This award supports a research development workshop for promising doctoral students to be held in conjunction with the 2014 ACM Conference on Computer Supported Cooperative Work (CSCW 2014). CSCW 2014 will take place in Baltimore, Maryland, USA on February 15-19, 2014 and will be attended by approximately 500 CSCW professionals from around the world. Research reports published in the CSCW Conference Proceedings are heavily refereed and widely cited.

The Doctoral Colloquium (DC) is a research-focused meeting of 12 selected Ph.D. candidates and a panel of 4-6 research mentors. Prior colloquia have helped launch the careers of many outstanding CSCW researchers. The colloquium seeks to (a) build a cohort group of new researchers who will then have a network of colleagues spread out across the world, (b) guide the work of the new researchers by having the experts in the research field give advice, (c) encourage and support the selection of CSCW research topics, (d) enable new entrants to the field to attend a key research conference, (e) illustrate the interrelationship and diversity of CSCW research, and (f) make the new entrants' experience at CSCW an intellectually stimulating and rewarding experience, encouraging them to return and submit papers, panels, demonstrations, and posters, to the conference. Key efforts of the organizers will encourage participation from diverse fields and underrepresented populations and universities.

The primary intellectual contributions of this project lie in close intellectual mentoring of the selected Ph.D. participants and the development of a cohort of future leaders within the CSCW community. Furthermore, bringing together these students with the faculty panel, both during the workshop and during the conference poster presentations, will support the development of interdisciplinary dialogs, creating an environment for exchange and conversation that will further enable progress on every project represented at the DC. The specific projects of students selected for the DC will generate additional intellectual contributions.

The consortium will have considerable benefits to society and the CSCW community, both short and long-term. In the short term, the panelists will provide significant feedback to the students participating in the DC. In the long-term, the expectation is that students who have participated in the DC will give back to the community by engaging as mentors to undergraduate students, and in the future to doctoral students. Furthermore, these exceptional students are anticipated to be among the future leaders of the CSCW community, and it is durable long-term benefit to that community to support them in their early research. These values of continuity and knowledge community will be explicitly expressed at the consortium.

This project will develop theory, algorithms, and example systems for motivating large numbers of autonomous individuals to solve problems that require coordination in the physical world. In these "physical crowdsourcing" systems, people make small contributions toward a larger collective problem, such as tracking animal species or air quality for citizen science projects or providing rides or package delivery in commercial applications. In these systems, opportunistically relying on people to do convenient parts of the problem causes incomplete solutions, while directing people to do inconvenient tasks requires high incentives. By modeling the timing and location of tasks, along with knowledge of people's routines, the project will develop algorithms that make use of ideas from decision theory to best decide when, where and to whom to suggest tasks, in a way that balances individual convenience with system needs. Doing this should increase people's willingness to participate, reduce the need to incentivize participation, and create more complete, timely, and accurate solutions to the collective problem. The investigators will develop several prototype systems that address practical problems such as package delivery and lost and found searches to demonstrate the effectiveness of their ideas; they will also release a toolkit that allows other people to use their work when designing their own physical crowdsourcing systems for solving both scientific and practical problems of interest to society.

The work will be carried out in three main phases. The first phase will develop "incentive chaining", a strategy to encourage opportunistic contributors to become more directable. Using an existing prototype for surveying tree species, the team will model people's opportunistic contributions and routines, then suggest relatively easy "nearby" tasks where others have contributed but where further work is needed, building people's interest and capacity to contribute. Once the nearby collaborative tasks are done, the system will direct these experienced and motivated contributors toward new areas and tasks, which will in turn become the "nearby" contributions that attract the next round of contributors. The second phase will develop decision-theoretic hit-or-wait algorithms to decide if and when to notify contributors about tasks, using a community-based lost and found application. Building on the modeling of behavioral routines, these algorithms will use sequential decision processes to estimate the maximize total value of tasks that contributors will complete, given individuals' history and current state, the system-determined value of nearby tasks, and the system's estimate of how likely the contributor would be to respond to a notification. The third phase will develop notification policies that satisfy the system's quality of service needs while minimizing disruption to contributors. To do this, the investigators will develop a supply management framework that considers the pool of available helpers, the current task demand and values, and a range of policies for making directed requests and evaluate it in the context of a peer-to-peer package delivery system.

Ensuring the accessibility of digital information and computational tools is crucial to enabling people with vision impairments to participate in all aspects of society, from education to career prospects to personal growth. This critical need is underscored by the fact that the millions of Americans with a visual disability are less likely to be employed or to pursue higher education than their sighted peers, and often have lower income levels as well. Although there has been tremendous improvement in information accessibility, the vast majority of research focuses on making content produced by others accessible. Yet, producing original content oneself (such as creative writing, podcasts, and music) provides valuable forms of employment, self-expression, and advocacy for people with vision impairments. This project will introduce novel accessible tools for creating text- and audio-based content, which will support the needs of blind students and professionals whether working independently or on teams with sighted collaborators. Outreach and dissemination of the work will take place through partnership with five community organizations for people with vision impairments, as well as online by means of open-source software and accompanying instructional materials. A "living laboratory for accessibility" will be created thal integrates research, education, and outreach through immersive learning experiences for undergraduate and graduate students, as well as opportunities for community members with disabilities to learn about and participate in research. The project has the potential to increase access to STEM-related educational experiences and career opportunities for people with vision impairments by bridging computation, engineering, and the arts. The research will also demonstrate an innovative approach to accessibility and content production that can be extended to other domains and applications.

Advancing accessible content production tools requires rethinking the way information is processed, rendered, and interacted with, which brings critical challenges in human-computer interaction, machine learning, and collaboration to the forefront of research. This project will contribute: (1) design guidelines and a taxonomy for accessible content creation tools; (2) novel algorithms to process text- and audio-based content, which enable new ways of presenting information while solving challenges relevant to machine learning, text analytics, and audio processing; (3) accessible interaction techniques that advance the ability of blind users to understand, navigate, and edit their work; (4) collaboration support features that will bridge gaps in collaboration systems and provide empirical and theoretical evidence of whether and how core collaborative processes operate in the context of mixed-ability teams; and (5) empirical evidence of how the developed systems support individual self-efficacy and collaboration during naturalistic use, thereby advancing inclusive online work and learning more broadly. Project outcomes will be assessed through a comprehensive evaluation strategy that captures individual, group, and societal level outcomes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.