Aniket Kittur - US grants

Virtual organizations are becoming increasingly important in driving production and innovation. Online production groups have the potential to transform the way that knowledge is produced and disseminated, but are often plagued with coordination problems that exceed those encountered by comparable co-located groups. Coordination is especially challenging for organizations engaged in complex, interdependent tasks typical of science and engineering. These groups often involve large numbers of contributors with tenuous and poorly defined relationships to the virtual organization and each other, who must coordinate to create coherent knowledge artifacts. For example, the volunteer editors who write Wikipedia, the online encyclopedia, must produce a collection of articles that are complete, accurate, coherent and readable. To solve these coordination challenges groups use a variety of mechanisms, including market bidding, standardization, strong leadership, and direct communication.

This project seeks to understand the effectiveness of particular coordination mechanisms for solving the coordination challenges faced by online production groups. The research will examine how online production groups use a variety of coordination mechanisms to solve problems of task assignment, temporal dependencies and fit between work outputs. It will examine the conditions under which these coordination mechanisms are successful in existing online knowledge production communities, such as Wikipedia and the thousands of other wikis that use a similar infrastructure. The effectiveness of these coordination mechanisms will then be tested in controlled online experiments, using artificial production groups recruited to synthesize information into coherent knowledge. This research extends theories of coordination to large scale virtual organizations, providing a greater understanding of how large groups of individuals can effectively work together online. The results will help virtual organizations to more effectively coordinate, engage participants in tasks that are important to the group, and coherently synthesize knowledge.

Social production communities can be powerful engines for harnessing the efforts of many individuals to produce valuable artifacts and knowledge. However, their success critically depends on members' ability to effectively contribute. As the size and complexity of the community grows, so do challenges to members' understanding of the content and collaboration. These challenges decrease the ability of the team to work together, and the quality of the work product.

The researchers propose partnering humans with intelligent interfaces that improve contribution effectiveness. They will create intelligent algorithms and interfaces that go beyond supporting people simply foraging for information to information farming, in which members of the community work together to plant the seeds of the information the community needs, nurture the growth of those seeds into valuable information, and weed out the information that detracts from the value of the farm.

The research is based on theories of human information processing, and will extend those theories to environments in which people are producing information. The researchers will explore new algorithms and interfaces based on the extended theories, and will carry out studies to understand how the theories work in practice.

Social production communities are economically and socially important. Open source software runs large parts of our economy; Wikipedia is revolutionizing knowledge production and consumption, providing free access to one of the largest bodies of knowledge gathered in human history. The proposed research will directly improve Wikipedia, and will contribute to understanding how social production communities work.

Crowdsourcing is a powerful way to marshal small contributions from large numbers of people to solve real-world problems. Success stories range from classifying craters on Mars' surface (ClickWorker) to labeling images (the ESP Game, now Google Image Labeler) to task marketplaces (Amazon's Mechanical Turk). This project moves towards a vision of crowdsourcing that extends it to support complex, creative, and interdependent tasks, and embeds it into computing systems as part of our everyday lives. The project will focus on two application areas for complex crowdsourcing: science journalism and software development.

The intellectual merits of the project include the uncovering of new scientific knowledge about how to model online crowd behavior, and the development of new methods and tools for using crowds as part of computer system designs, particularly for complex, interdependent, real time work. The project will also show that these methods can be used for real-world problems.

The potential broader impacts include those specifically having to do with the two application areas, which could have significant impacts on society. Crowdsourcing science journalism will directly involve citizens in the process of science dissemination, making scientific information more accessible to the general public, and promoting greater awareness of science and the scientific process. Crowdsourcing software development can transform the way that software is created, lowering barriers and broadening participation in open source software development, and helping larger masses of people use and improve their programming skills. Other impacts will flow from the researchers' plans to publically share the infrastructure that they develop to facilitate complex crowdsourcing in many other areas. They also plan to integrate their research results into undergraduate courses.

This project will develop web-based knowledge-building environments for the collaborative creation of information landscapes: interactive visualizations that support the sensemaking individuals engage in online, and capture their efforts for the benefit of others who come after them. To guide and motivate the design of these environments, the research builds on theories of sensemaking, which describe the nested and parallel loops through which individuals seek out, analyze, and understand information, grounded in a rich history from organizational behavior, social and cognitive psychology, and human-computer interaction. It will extend theories of individual sensemaking to the situation in which an individual's processing of information for themselves is consumed by others, whose processing in turns improves the sensemaking of those coming after them - what can be called the "distributed sensemaking cycle."

Examining distributed sensemaking in a way that is both rigorous and environmentally valid is a challenging prospect, so this project will take a multi-stage approach involving laboratory studies to characterize the distributed sensemaking process and iteratively develop interfaces; "virtual lab" experiments harnessing crowdsourcing to evaluate these processes and interfaces at a larger scale while bootstrapping the system's value; and controlled field trials to test theories and interfaces in environmentally valid settings.

In order to capture the benefits and costs to both the producer and the consumer, the research will employ an experimental framework that elicits both of their perspectives. The general approach involves the producer using an interface for a sensemaking task and the consumer doing the same task but starting with the results of the producer's work. This approach will iteratively develop interfaces that help individuals forage for information, integrate the results of their foraging into information landscapes, and convey the judgments, decisions, and work they engaged in during the process to others.

The results of this research will advance scientific understanding across a variety of domains, including sensemaking, collaboration, schema induction, and interface design. The research has the potential to improve the efficiency of knowledge work, the training and practice of scientists, and the effectiveness of education. The tools developed in this research will be incorporated and evaluated in educational practice, and will become the center of a community linked to several existing knowledge bases.

The goal of this research project is to help people make sense of large graphs, ranging from social networks to network traffic. The approach consists of combining two complementary fields that have historically had little interaction -- data mining and human-computer interaction -- to develop interactive algorithms and interfaces that help users gain insights from graphs with hundreds of thousands of nodes and edges. The goal of the project is to develop mixed-initative machine learning, visualization, and interaction techniques in which computers do what they are best at (sifting through huge volumes of data and spotting outliers) while humans do what they are best at (recognizing patterns, testing hypotheses, and inducing schemas). This research addresses two classes of tasks: first, attention routing -- using machine learning to direct an analyst's attention to interesting nodes or subgraphs that do not conform to normal behavior. Second, sensemaking -- helping analysts build in-depth representations and mental models of a specific areas or aspects of a graph. Evaluation of the tools will involve both controlled laboratory studies as well as long-term field deployments.

As large graphs appear in many settings -- national security, intrusion detection, business intelligence (recommendation systems, fraud detection), biology (gene regulation), and academia (scientific literature) -- the potential benefits of new tools for making sense of graphs is far reaching. Project results, including open-source software and annotated data sets, will be disseminated via the project web site (http://kittur.org/large_graphs.html) and incorporated into educational activities.

This social computing research will build a scientific foundation for distributed analogical innovation that integrates cognitive science, social science, design, and computer science. The results will contribute back to theory in these source disciplines as well, including theories of analogy, coordination, distributed innovation, scientific discovery, crowdsourcing, and collective intelligence. Innovation in science and technology is often driven by analogy, and opportunities for finding fruitful analogies are exploding with the increased online availability of repositories of ideas ranging from scientific papers to product ideas to the diverse text and video resources. However, our ability to process this deluge of information to find and use analogies is severely bottlenecked by individual cognitive limits, as the speed and capacity with which individuals can learn and explore new domains have not kept up with the rapid growth in online information from which analogies can be discovered. Furthermore, people are hampered by cognitive biases that prevent them from seeing and applying important parallels to solve comparable problems in different domains. Instead of relying on a single individual to find and apply an analogy, this research will develop methods for distributing analogical processing across multiple individuals. The development of improved innovation processes could accelerate technological innovation and scientific discovery, with corresponding benefits to the economy and to science.

The general approach will be to conduct empirical studies to rigorously explore and characterize the benefits and limitations of distributed analogical innovation, and then to generalize these findings in the context of two real-world innovation communities. Distributing innovation tasks to different individuals could have a number of significant advantages. First, increasing the number of people involved could increase the capacity to find and use analogies across many domains. Second, changing the representations of problems and solutions could help innovators see the relevance of solutions from distant domains and reduce problems with fixation, in which people are overly influenced by surface features. Third, by distributing the steps involved in analogy to different people, alternative innovation paths could be explored effectively in parallel. Fourth, disaggregating the innovation pipeline opens up participation to many more people; one need not be an expert in a problem domain, for example, to find analog solutions in remote domains. Fifth, in contrast to innovation contests that reward only a small number of contest winners, the sequential method to be used in this research allows people to build on each other's work and doesn't waste the labor of the vast majority of participants.

This PFI: AIR Technology Translation project focuses on translating distributed sensemaking platforms to fill the need for understanding complex online information.   Distributed sensemaking is important because people spend over 70 billion hours on gathering information online each year for tasks ranging from making sense of their medical symptoms to learning a new scientific field. However, all the work done in gathering and structuring this information is lost, with each person having to start from scratch every time.  Instead, this project will develop a new information gathering platform in which the work that individuals do for themselves can be captured and made useful for others with similar needs. This distributed sensemaking approach will go beyond existing search engines, which focus only on providing relevant web pages, to helping users build rich information landscapes that capture information, evidence, judgments and perspectives across multiple sources and sites.

This project addresses the following technology gaps as it translates from research discovery toward commercial application. Although the distributed sensemaking approach can be tremendously useful in accelerating a person's knowledge of a new domain and improving their decision-making, a fundamental barrier to this approach being adopted in the real world is that without enough initial users and the processing they provide, using the system is perceived as less valuable than just starting from scratch. This proposal aims to develop a new platform that will be useful to individuals themselves in managing complex searches, while simultaneously capturing the rich sensemaking they do while searching. In the short term, this platform will help people find, manage, and synthesize information for themselves from multiple web pages and for multiple tasks on a variety of devices ranging from smartphones to desktops. In the long term this platform will be the foundation to kickstart a virtuous cycle of distributed sensemaking, moving the web from a pattern of increasing fragmentation to increasing synthesis as more people use it. In addition, personnel involved in this project, including graduate students and undergraduates, will receive innovation experiences through iterative, user-centered development and testing opportunities.

Comparisons between two things are useful to support creative thinking and reasoning. Many important breakthroughs in science and technology were driven by analogy: observing water led the Greek philosopher Chrysippus to speculate that sound was a wave phenomenon; an analogy to a bicycle allowed the Wright brothers to design a steerable aircraft. Opportunities to find analogical breakthroughs are exploding with the increased availability of online repositories of ideas ranging from scientific papers to patents to the entire web. This project will use computational tools to facilitate searching for analogies drawn from one domain to help with innovative thinking and reasoning in another domain. The goal is to dramatically accelerate innovation and discovery across domains as diverse as science, humanities, mathematics, law, and design.

The planned research bridges the gap between the power of large-scale text mining approaches, which excel at detecting surface similarity, and the depth of human cognition, which is currently unsurpassed at detecting deep analogical similarity. Investigators will explore how representations with weaker structures are robust to issues of language complexity, hierarchies of purposes, and levels of abstraction that are present in real-world documents like research papers and R&D documents. Using these representations, investigators will build computational tools enabling users to connect problems in one field with solutions from another field based on deep structure. Results and algorithms could spur the development of new types of machine-learning techniques that focus on deep structure, and may contribute back to theory in the fields of creativity, problem solving, and innovation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Programming is a critical skill that is vital for the future of work and having a globally competitive workforce. While there are many resources available for programmers to learn the details for writing code, an increasing amount of the time all programmers spend is not on writing code but instead on choosing among and adapting the growing amount of existing code and libraries available to them. One study reported that the most frequent programmer activity is searching for and trying to understand unfamiliar code, and more than 30% of all searches are for determining which APIs to use and how to use them. However, after each sense-making episode in which a programmer gains knowledge for themselves, their work is essentially lost, with no one else benefiting. Although there are many tools to help programmers find the answers, there are very few tools to help programmers make use of the knowledge gained performing the task, or share that knowledge with others. Capturing the work that programmers do in foraging, navigating, and organizing code-relevant information could significantly benefit later programmers interested in similar information. By referencing the captured knowledge from the resulting code, this can provide design rationale for why the API is used that way, which is one of the most often missing pieces of documentation. Furthermore, by making it easier for programmers to build off one another's knowledge, this proposed work has the potential to reduce common security vulnerabilities that arise from programmers not learning from others' mistakes, leading to more secure and correct code.

In this research the PI aims to help the initial programmer collect, navigate, and organize knowledge to meet their goals, while capturing this knowledge and making it useful for later programmers with similar needs. This project studies the sense-making processes that programmers engage in while searching for and organizing knowledge for themselves, as well as studying which work that they do is useful for others. This project investigates how programmers spend their time searching for and making sense of complex information for themselves in order to accomplish their goals, including choosing among different APIs or methods within an API, adapting code snippets found on the Internet to meet their needs, or trying to learn unfamiliar code to fix an error or add a new feature. When performing tasks like these, programmers continually are making hypotheses, proposing questions, and discovering answers, both about the details as well as the meta-level questions such as the design rationale of why the decisions were made. These studies will inform the design, development, and evaluation of tools to support both the initial and later programmers. This research has the potential to significantly accelerate the speed at which programmers can create correct code by helping them gain relevant knowledge faster.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Many rural areas in the United States face a lack of economic opportunity. The future of work can bring opportunities for rural and urban marginalized communities through online work and the gig economy. However, work on current platforms is often low-level labeling work offering few opportunities for advancement. It is often intended to train Artificial Intelligence to automate this work away, instead of training workers. The proposed project aims to uplift workers and improve the marketplace for online work so that digital work may help with the economic recovery of regions whose traditional industries have left. This project aims to develop sustainable methods for transitioning workers to high-skilled and creative digital jobs that are unlikely to be automated in the near to medium term future. Crowd work can be transformed to not only improve the work product for the employer, but also to help the worker move along the career paths necessary for the future of work. The project team from four universities, Carnegie Mellon U., West Virginia U., Pennsylvania State University and University of Pennsylvania has partnered with local institutions to provide workers training to perform progressively more advanced digital work, while earning money. The vision of the project is to scaffold workers through basic computer fluency, working with AI tools, and finally innovation and creativity skills. This work is in collaboration with a rural partner (Rupert Public Library, in Rupert, WV) and urban partner (CommunityForge in Wilkinsburg, PA) and also benefits from a partnership with Bosch Inc. in Pittsburgh, ConservationX Labs in Washington DC, and the State of West Virginia.

The proposed research addresses a fundamental challenge in that those who most need to develop skills to gain higher paying jobs cannot afford the unpaid time spent in training needed to develop them. Accomplishing this vision will require solving the following core research questions: (i) How can one best support the marginalized workers in their transition to online work?, (ii) How can Artificial Intelliegnce tools augment workers, rather than displace them?, (iii) How can tools be designed to help workers build skills and creativity for work that is unlikely to be automated in the future?. This project has the potential to make advances across a variety of interrelated fields including crowdsourcing, Artificial Intelligence, Human Computer Interaction, Cognitive Science, Learning Science, Sociology and Economics. Simultaneously enabling both improved work outcomes as well as skill development in crowd work will require the development of models of workers, skills, and their trajectories at a more nuanced level. Enabling workers to collaborate with Artificial Intelligence will require new human-computer interaction paradigms. Supporting creativity and the development of new skills will require the exploration of new organization and coordination structures. By grounding the investigations in real world contexts, the research aims for generalizable knowledge that can lay a foundation for research on the future of crowd work at the human-AI frontier

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.