Christian D. Schunn - US grants

The Learning Research and Development Center at the University of Pittsburgh will conduct a longitudinal study in which researchers will create a three-year sequence of model-centered instruction in the context of the Model-Assisted Reasoning in Science (MARS). MARS current topics will be extended from sixth through eighth grades. The project seeks to: (1) understand how working external models support content and process learning, (2) develop an evaluation model to tap strengths and weaknesses of different kinds of external models, and (3) identify pedagogical strategies that elicit and support model-assisted reasoning. Student content knowledge and process skills will be measured through different test formats that include paper-and-pencil (TIMSS, NAEP, and Test of Scientific Reasoning items), written tests, class work, and classroom computer exercises. Student motivation will be measured at the beginning of each year. A small sample of students will be interviewed and given some transfer tasks twice a year. Interviews will focus on two aspects: properties of the different model types and student's metacognitive understanding of the function of models in science. Information on classroom implementation will be collected by direct observation, videotapes, and interviews with teachers. Results of the study are expected to help extend theories of model-based reasoning and its applicability in classrooms.

This award provides funding for a 3 year continuing award to support a Research Experiences for Teachers (RET) Site program at the University of Pittsburgh entitled, "Bringing Innovative Design into Urban High Schools on a Sustainable Basis: University of Pittsburgh Design Team RET Site," under the direction of Dr. Michael R. Lovell. The mission of this RET site program is to implement several of the University of Pittsburgh's innovative design research activities at the high school level in an effort to foster creativity and promote interest in science, technology, and math (STEM) subjects, particularly for underrepresent pre-college students. Two well recognized units of the University of Pittsburgh are joining together to attain this mission-the Swanson Center for Product Innovation (SCPI) and the Learning Research and Development Center (LRDC). To gain an additional real-world industry perspective, Westinghouse Electric Corporation (WEC) will also significantly contribute to the propsed research program by providing leadership, projects and financial support for the RET activities.

Co-funding is being provided by the NSF Partnerships for Innovation Program (PFI).

How do scientists and engineers discover and innovate? The creative process is often portrayed as one shrouded in mystery, and the domains of science and engineering are no exception. Imagine the stereotype of the eccentric professor, for instance, staring intently at a blackboard full of equations, who then suddenly sees the answer that had been hidden for months or even years. Or imagine a team of engineers going out for lunch to take a break from an impasse they have hit, and in the middle of casual conversation they come upon a novel approach to the problem, one that elegantly gets around the impasse. These scenarios depict the births of discoveries and innovations that, with nurturing and perseverance, can grow to have long-lasting impacts on the fields of science and engineering, and the world at large.

The National Science Foundation is sponsoring a workshop centered around the creative process as it pertains to science and engineering. The workshop will bring together cognitive scientists, social psychologists, and engineers to share their research in an effort to identify some of the factors that contribute to discoveries and innovations. In doing so, the workshop will help to open up a cross-disciplinary dialog with the goal of producing useful knowledge for making investments that advance the frontiers of science and engineering.

Engineering design is not a purely mental process. Successful design engineers move from ideas to completed designs using artifacts and tools. Over the past several decades, the number of tools and artifacts available to engineers has become virtually limitless. These tools include drawing programs, quantitative modeling software, sketch paper, CAD programs, and prototyping facilities. Despite their importance for supporting creativity and innovation, little is known about the role of tools in supporting the cognitive processes of innovative design.

With support of the National Science Foundation, Dr. Schunn and Dr. Novell will investigate the roles of tools and artifacts in the creation of innovative designs. Their research team combines the strengths of cognitive science theories and methods with best practices in innovative design education. The investigators will conduct a large-scale experiment in which approximately 2500 hours of video data will be collected. These data will capture the design activities of about 50 undergraduate and graduate-level engineering design teams. The data will be strategically sampled in order to establish promising directions for more in-depth coding efforts on the nature of cognitive processing in conjunction with design tools. The goal is to identify the combinations of tools, artifacts, and cognition that lead to more innovative design outcomes.

Across the sciences, computational modeling is one the most widely-used methods for linking different levels of analysis, to show how processes at one level can interact to produce emergent system behavior at another level. Within the cognitive science, computational modeling has been used to link neural, mental, and behavioral levels of analysis, for instance. Computational modeling has also served as a common ground for drawing links across the various disciplines that comprise the cognitive sciences, such as cognitive neuroscience, linguistics, computer science, psychology, anthropology, philosophy, and education. Computational modeling has clearly become a valuable and oftentimes needed skill in the cognitive sciences, yet there are barriers to acquiring and practicing this skill for students and junior investigators entering the field. To encourage the practice of computational modeling, the National Science Foundation will contribute to prizes and tutorials for computational modeling submissions to the 2006 Meeting of the Cognitive Science Society. The prizes and tutorials will be aimed at getting students interested and involved in computational modeling, and to provide some initial training that might spur them to seek further training through their institutions and other resources.

Across the sciences, computational modeling is one the most widely-used methods for linking different levels of analysis, to show how processes at one level can interact to produce emergent system behavior at another level. Within the cognitive science, computational modeling has been used to link neural, mental, and behavioral levels of analysis, for instance. Computational modeling has also served as a common ground for drawing links across the various disciplines that comprise the cognitive sciences, such as cognitive neuroscience, linguistics, computer science, psychology, anthropology, philosophy, and education. Computational modeling has clearly become a valuable and oftentimes needed skill in the cognitive sciences, yet there are barriers to acquiring and practicing this skill for students and junior investigators entering the field. To encourage the practice of computational modeling, the National Science Foundation will support student training in computational modeling at the 2007 Meeting of the Cognitive Science Society. Support is aimed at getting students interested and involved in computational modeling, and providing tutorials that might spur them to seek further training opportunities through their institutions and other resources.

Innovation and discovery involve individuals working successfully together in teams. It is critical for the Science of Science and Innovation Policy to understand how the cognition of individuals, the direct source of novel ideas and critical decision making, is impacted by social teamwork variables. Prior research has typically studied social teamwork variables in isolation or individual cognition variables in isolation. To know how to intervene to increase engineering and scientific output, the relationships between the two must be known, or else we might improve one at the cost of hurting the other, which likely would have no net improvement in final scientific or engineering productivity. The current project examines a very large quantity of video data collected from a recent highly successful case of science and engineering, the Mars Exploration Rover, which both wildly exceeded engineering requirements for the mission and produced many important scientific discoveries. Yet, not all days of the mission were equally successful. From this video record, the project traces the path from the structure of different subgroups (such as having formal roles and diversity of knowledge in the subgroups) to the occurrence of different social processes (such as task conflict, breadth of participation, communication norms, and shared mental models) to the occurrence of different cognitive processes (such as analogy, information search, and evaluation) and finally to outcomes (such as new methods for rover control and new hypotheses regarding the nature of Mars).

Another critical factor for Science of Science and Innovation Policy is to examine both divergent thinking and convergent thinking. Innovation rarely happens unless new ideas are considered. But progress will not happen unless the best ideas among the proposed set are ultimately selected. To know how to best intervene to improve discovery and innovation, progress must be made on finding out when to intervene, which likely depends upon whether divergent or convergent thinking is currently required. A number of prior inconsistent research results likely resulted from a failure to separately consider divergent and convergent thinking. The current project examines both elements to build a much more complete model of how cognitive and social variables come together to produce new and successful engineering innovations and scientific discoveries.

Broader Impacts: The US is facing serious challenges in the fields of science and technology. Innovation must be harnessed to generate new products, create employment opportunities, and strengthen the national economy. It is vital that the flourishing of science and engineering teams be examined with the same rigor as other important human endeavors. This project also has implications for science and engineering education: as ways of composing, structuring, and instructing teams are examined, suggestions for pedagogy will be formulated based on empirical findings.

The U.S. is facing serious challenges in the fields of science and technology and our future engineers must use innovation to generate new products, create employment opportunities, and strengthen the national economy. Furthermore, the existing connections between cognitive science and engineering are nationally quite small. This study provides a much needed diverse set of investigators to bridge knowledge across these areas.
Engineering design is a rich interplay of physical and mental. Successful design engineers move from ideas to completed designs using artifacts and tools. Over the past several decades, the number of tools and artifacts available to engineers has become nearly limitless. These tools include drawing programs, quantitative modeling software, sketch paper, computer aided design programs, and prototyping facilities. Despite their importance for supporting creativity and innovative ideas, little is presently known about how tools differentially support innovative design. A lack of knowledge how these tools impact innovative design limits improvements to tools and practical knowledge of how and when to use those tools in design.
The goal of the current study is to begin to build a fundamental understanding of the cognitive processes underlying the role of tools and artifacts in the innovative design process, by combining strengths in cognitive science research and innovative design education (Swanson Center for Product Innovation) at the University of Pittsburgh. A large-scale experiment that examines the ways in which artifacts and tools contribute to innovative design is conducted. This experiment is used to collect a massive database of design activities, consisting of approximately 3,000 hours of video from approximately 60 undergraduate and graduate-level engineering design teams using cyber-infrastructure for video collection leveraged through this grant. The video is then strategically sampled to unpack the causal path from design/tools artifacts in the environment, to core cognitive processes underlying design, to dimensions of design creativity, to the ultimate success of the designed object. In addition, new engineering design innovativeness metrics are developed, validated, and refined. The merit of this study lies within the fact that such a comprehensive study on the learning of design tools and artifacts will substantially expand the understanding of the fundamental processes involved in this important, but often overlooked field.
At the completion of this study, it is expected that the knowledge gained will allow the investigators to lead the development of a new suite of design tools and strategies for supporting practicing engineers and educating engineering students. These tools and strategies could have far reaching implications as they could initiate substantial changes in design practice and engineering design education. This study also involves an enormous data collection effort. The resulting video database will provide volumes of data on the role of tools and artifacts in innovative design, only a small portion of which is to be analyzed during the grant period. This video database will foster considerable follow-up analyses for years to come. The resulting video database will be disseminated throughout the academic community.

This award provides funding for a 3 year continuing award to support a Research Experiences for Teachers (RET) in Engineering Site program at the University of Pittsburgh entitled, "Connecting Research and Teaching Through Product Realization: The Pittsburgh Quality of Life RET Site," under the direction of Dr. Michael Lovell.

This program is a renewal of a successful RET Site, which retains the best elements of the existing RET, integrates new elements of best practicces observed in other RET programs, and embraces a new Quality of Life (QoL) engineering research theme. A total of 33 high school science teachers (11 per year for three years) will be recruited from high needs urban high schools in the Pittsburgh area. They will spend 8 weeks in the summer working in engineering labs performing fundamental scientific research and then translate that research into innovative products, involving topics related to Quality of Life Technology. The major research thrust in the QoL theme will focus on innovations in computation, robotics, machine learning, communication, and miniaturization technologies that transform the lives of people with reduced functional capabilities. Other program activities will include redesigning design-based-learning (DBL) units that will be implemented in their classrooms, workshops and classroom visits by faculty mentors.

The site will directly impact 33 teachers and 4,500 students from high need urban high schools in the Pittsburgh region, with a particular focus on high schools comprised almost entirely of minority and low socio-economic status students. By focusing on QoL, the RET Site will improve awareness and develop technology for a growing segment of the population that is often overlooked--people with reduced functional capabilities due to aging or disability.

The research objective of this award is to improve understanding and capabilities in concept generation through design by analogy methods. The proposed approach, a collaboration between the disciplines of cognitive psychology, engineering design and computer science, is to provide new tools for design based on a representation that associates functional and geometric information, combining a linguistic search for functional similarity with a multi-level search for geometric similarity to automatically identify and present analogies to the designer. The initial application for the Verrocchio Project is the design of prosthetic and orthotic devices for persons with disabilities, a domain that is ripe for innovation. The initial search space is the USPTO utility patent repository. Deliverables will include: (1) means to more effectively generalize design problems through functional descriptions; (2) the ability to search for analogical solutions with alternative functional representations; (3) ways to search for geometric similarities across a set of functional analogies; and (4) the ability to produce a tractable set of analogies for use by the designer.

If successful, the results of this research will have five broad and transformative impacts. The first impact arises from the key product of the research: a method for systematically and automatically identifying analogies. This tool will have broad applicability across many product domains, and will improve the efficiency of the search for innovative products in those domains. Second, the research will advance the foundation for use of analogies in engineering design methods. Third, because the research will result in a computer implementation of the method, the work will add to the growing cyber infrastructure of the country. Fourth, the interdisciplinary nature of the research ensures cross-fertilization of theory and research techniques in each of the collaborators? disciplines. This interdisciplinarity will enhance the probability of broad adoption of the approach. Fifth, instructional materials will be developed for training students in the approach. These instructional materials will be freely available to instructors at other institutions, encouraging broader adoption of the approach.

Understanding how organizations promote innovation is a key element of advancing the science of innovation, and hence the science of innovation policy. Data on innovation inputs, innovation processes, and innovation outputs are increasingly being captured and stored electronically. A number of fundamental bottlenecks to using these data to advance social science research exist due to unsolved issues of privacy, data integration, and data quality. The core scientific challenge is how to make such real-world, large-scale data available to researchers to nurture innovation and perform valid experimentation, while maintaining data privacy. Fortunately, computer scientists have been developing a variety of techniques and building new tools that manage large data sets in ways that can potentially help in supporting and measuring innovation activities.

This workshop brings together social scientists, the users of data on innovation, together with computer scientists, the creators of new tools for collecting data while protecting privacy concerns. The workshop includes leading computer scientists with specialties in data management, data mining, security/privacy and social networks as well as social/organizational scientists, such as economists, sociologists, psychologists and anthropologists.

The focus of the workshop is to identify emerging major challenges in this interdisciplinary area. Three different types of data critical to the study of innovation are the focus of study: third party data, such as U.S. Census data, patendt databases, NSF funding data or citation databases; detailed insider data such as internal communications, team video, or team documentation; and broader insider data such as cross-firm surveys.

The broader impact of the workshop is substantial. First, an enhanced empirical basis for studying innovation is necessary to guide policy decisions. Second, the workshop participants represent a broad variety of disciplines, which creates a broader interdisciplinary network of study. Finally, the attendance of graduate students advances the goal of training a new cohort of researchers in the field.

Research in biology has become increasingly mathematical, but high school courses in biology use little mathematics. To address this concern, this project will develop three replacement units for biology and refine them through classroom testing. The units will be models of STEM integration by using the important concepts of proportional reasoning and algebraic thinking and engineering re-design to address big ideas in science while also promoting the learning of 21st century skills. The materials build on existing work on the use of model eliciting activities and focus science and technology instruction on high-stakes weaknesses in mathematics and science. They address the scaling issue as part of the core design work by developing small units of curriculum that can be applied by early adopters in each context. The materials will undergo many rounds of testing and revision in the early design process with at least ten teachers each time. The materials will be educative for teachers, and the teacher materials and professional development methods will work at scale and distance.

Learning of science content will be measured through the use of existing instruments in wide use. Existing scales of task values, achievement goals and interest are used to measure student motivation. The work performed is guided by a content team; a scaling materials team; a scaling research team; the PI team of a cognitive scientist, a robotics educator, and a mathematics educator specializing in educational reform at scale; and the summative evaluation team lead by an external evaluator.

There is great interest in understanding whether integrated STEM education can interest more students in STEM disciplines. The focus on mathematics integrated with engineering in the context of a science topic is interesting and novel and could contribute to our understanding of integrating mathematics, engineering and science. The development team includes a cognitive scientist, a mathematics educator, teachers and scientists. The issues and challenges of interdisciplinary instruction will be investigated.

The Robot Algebra Project creates three scalable, middle school level units for use in informal settings. The units are designed around fundamental robot movement concepts but emphasize proportional reasoning - a big idea in mathematics. There are over 12,000 FIRST Lego League teams across the U.S. that purport to use robots as a motivator to engage students in STEM. However, most of the time the students use guess and check procedures thwarting the opportunity to learn STEM content. The units being developed build upon model eliciting activities, project-based learning and mathematics education to specifically improve student understanding of a few key mathematics concepts. The programming of robots is scaffolded so that students concentrate on the mathematics. Rather than only doing hands-on activities, the students also produce toolkits for other students to engage in similar experiments. Paper- based word problems are developed to bridge the mathematics learned in the context of robotics to generalized mathematical problem-solving strategies. Professional development is provided both face-to-face and through webinars to early adopters who are also trained to provide professional development to others. Materials to supplement the professional development are produced to support teachers and informal educators understanding of the rationale, the agenda, the mathematics and the perspectives that underlie the student materials as well as to also support them in anticipating student responses to the tasks. The materials can be updated online.

Pre and post tests against a control group in standard robotics programs are used to provide formative and summative evaluation. Ten students are interviewed each year about how the experiences affect their career choice. The scalability of the use of the units is measured through observation of and interviews with teachers. In the third year the units will be used in beta sites.

The fundamental goal of the current proposal is to improve middle school level student's algebraic reasoning ability, specifically their understanding of proportionality. A secondary but critically related goal is to improve informal educator's pedagogical content knowledge with respect to algebraic reasoning ability.

The PIs are investigating the design of intelligent tutoring systems (ITSs) that are aimed at learning in unstructured domains. Such systems are not able to do as much automatically as ITSs working in traditionally narrow and well-structured domains, but rather they need to share responsibilities for scaffolding learning with a teacher and/or peers. In the work proposed, the three PIs, who share expertise in automated natural language understanding, intelligent tutoring systems, machine learning, argumentation (especially in law), complex problem solving, and engineering education, are integrating intelligent tutoring, data mining, machine learning, and language processing to design a socio-technical system (people and machines working together) that helps undergraduates and law students write better argumentative essays. The work of helping learners derive an argument is shared by the computer and peers, as is the work of helping peer reviewers review the writing of others and the work of learners to turn their argument diagrams into well-written documents. Research questions address the roles computers might take on in promoting writing and the technology that enables that, how to distribute scaffolding between an intelligent machine and human agents, how to promote better writing (especially the relationship between diagramming and writing), and how to promote learning through peer review of the writing of others.

This project is bringing together outstanding researchers from a variety of different disciplines -- artificial intelligence, law education, engineering and science education, and cognitive psychology -- to address an education issue of national concern -- writing, especially writing that makes and substantiates a point -- and to explore ways of extending intelligent tutoring systems beyond fact-based domains. It fulfills all aims of the Cyberlearning program -- to imagine, design, and learn how to best design and use the next generation of learning technologies, to address learning issues of national importance, and to contribute to understanding of how people learn.

This award provides funding for a 3 year continuing award to support a Research Experiences for Teachers (RET) in Engineering and Computer Science Site program at the University of Pittsburgh entitled, "Connecting Research and Teaching Through Product Innovation: Quality of Life Technology RET Site under the direction of Dr. Jonathan L. Pearlman.

This RET Site is a renewal of a successful RET Site program hosted by the University of Pittsburgh, which will continue to draw upon the aggregate strengths in the Learning Research and Development Center (LRDC) and the Quality of Life Technology (QoLT) Engineering Research Center. This site will recruit a total of up to 39 science and math teachers, as interdiscipliary pairs from high needs urban high schools, who will complete:
1) A 12 week experience in engineering labs in across-district teams of 4 to perform fundamental scientific research and then translate that research into innovative products, involving QoLT topics.
2) A course on product innovation that includes design, development, and evaluation of technolgies in addition to the processes of bringing these products to market.
3) A redesign of design-based learning (DBL) units originally developed in the LRDC, and modified via iterative design by RET teachers over time to incorporate product development economics and other math concepts related to innovation.
4) A comprehensive professional development series providing teachers with classroom implementation and assistance of DBL units.

This site will directly impact approximately 39 teachers and 400 underrepresented students including those from high need urban high schools in the Pittsburgh region, with a particular focus on high schools comprised almost entirely of minority and low socio-economic status students, and girls. In addition, by focusing on QoLT, this site improves awareness and develops technology for a growing segment of the U.S. population that is often overlooked-people with reduced functional capabilities due to aging or disability.

This is a project to improve understanding of practices critical to the design of curricular materials for implementation in a broad range of educational contexts. Three organizations - TERC, the University of California-Berkeley's Lawrence Hall of Science, and the University of Pittsburgh's Learning Research and Development Center - will collaborate to explore and codify practices that enhance the success of efforts to design K-12 science curriculum materials for large-scale implementation. Investigators from these three organizations will conduct and synthesize results from a series of retrospective and live-design practice, broad and 'deep dive' studies, with the goal of articulating a conceptual model of educational design for large-scale use. Of particular concern are the processes and strategies designers employ to address key challenges to producing curricular materials capable of having meaningful impacts on large numbers of learners (e.g., to achieve deep understanding and rich performance, to connect to and leverage diverse social and cultural experiences, and to facilitate implementation in diverse and resource-limited settings). These issues will be explored from a variety of perspectives, including: interviews with designers and document reviews to identify structural project characteristics that appear to be empirically associated with scaling success; retrospective case studies to identify salient features and lessons learned from more and less successful large-scale design initiatives for science education; and deep dives (involving participant-observation, interviews, focus group discussions, and document analysis) into sustained design practices over an extended period to explore how design teams address key design challenges while developing educational materials for large-scale use.

Computational and algorithmic thinking are new basic skills for the 21st century. Unfortunately few K-12 schools in the United States offer significant courses that address learning these skills. However many schools do offer robotics courses. These courses can incorporate computational thinking instruction but frequently do not. This research project aims to address this problem by developing a comprehensive set of resources designed to address teacher preparation, course content, and access to resources. This project builds upon a ten year collaboration between Carnegie Mellon's Robotics Academy and the University of Pittsburgh's Learning Research and Development Center that studied how teachers implement robotics education in their classrooms and developed curricula that led to significant learning gains. This project will address the following three questions:

1.What kinds of resources are useful for motivating and preparing teachers to teach computational thinking and for students to learn computational thinking?
2.Where do teachers struggle most in teaching computational thinking principles and what kinds of supports are needed to address these weaknesses?
3.Can virtual environments be used to significantly increase access to computational thinking principles?

The project will augment traditional robotics classrooms and competitions with Robot Virtual World (RVW) that will scaffold student access to higher-order problems. These virtual robots look just like real-world robots and will be programmed using identical tools but have zero mechanical error. Because dealing with sensor, mechanical, and actuator error adds significant noise to the feedback students' receive when programming traditional robots (thus decreasing the learning of computational principles), the use of virtual robots will increase the learning of robot planning tasks which increases learning of computational thinking principles. The use of RVW will allow the development of new Model-Eliciting Activities using new virtual robotics challenges that reward creativity, abstraction, algorithms, and higher level programming concepts to solve them. New curriculum will be developed for the advanced concepts to be incorporated into existing curriculum materials. The curriculum and learning strategies will be implemented in the classroom following teacher professional development focusing on computational thinking principles. The opportunities for incorporating computationally thinking principles in the RVW challenges will be assessed using detailed task analyses. Additionally regression analyses of log-files will be done to determine where students have difficulties. Observations of classrooms, surveys of students and teachers, and think-alouds will be used to assess the effectiveness of the curricula in addition to pre-and post- tests to determine student learning outcomes.

Innovation fundamentally begins with a good idea. But where do good ideas come from? Much research suggests that innovative breakthroughs are often inspired by past experience: things and ideas that one has interacted with in the world. However, the same experiences that can inspire innovation can also can constrain or harm innovation through focus on previously unsuccessful solutions. This project tests principles for guiding interactions with sources of inspiration to maximize their benefits and minimize their pitfalls. In particular, it focuses on the role of conceptual distance of sources. The following questions are the focus of this work: 1) Are good ideas built mainly on sources that are closely related to the problem (e.g., building on existing recycling efforts to address the problem of people throwing away electronics), or are they most often inspired by sources that are from distantly related domains (e.g., being inspired by how burrs cling to a dog's fur when designing Velcro)? 2) When considering multiple sources, should one try to ensure that the sources are similar to each other (i.e., deeply exploring one direction), or should one consider diverse sources?

Intellectual Merit
Innovation researchers and practitioners have formulated recommendations for these questions, but the scientific evidence for these recommendations is incomplete because it is based on small numbers of case studies or involved toy problems solved much more quickly than real-world problems. This project expands and improves the evidence base for understanding the sources of innovation by analyzing thousands of solutions to complex innovation challenges (e.g., increasing accessibility in elections, revitalizing struggling urban areas) posted on an online crowd-sourced innovation platform that required contributors to post sources of ideas. To analyze this evidence, it combines computer algorithms for automatically processing the content of text along with statistical algorithms for extracting meaning across thousands of ideas. The quality of the ideas are coded by experts for quality and novelty, and then the algorithms are used to statistically test the relationship between both source distance and source variety on ideation success.

Broader Impacts
The results of this work will have immediate implications for scientific theories of innovation, and are relevant to the design of systems for creating new innovations (e.g., online crowd-sourced innovation platforms, creativity support systems and methods). Further, the novel methods employed in this work, as well as the data generated, will be shared with other innovation researchers to facilitate more and better investigations into the science of how innovation happens. Ultimately, this work contributes to a solid scientific research base that can support innovators (both existing and aspiring) and policy makers in their efforts to address the most pressing and difficult problems facing the world today.

The University of Pittsburgh and Carnegie Mellon University will test and refine a Theory of Badges applied to Computer Science Education. Given that badges have recently attracted a great deal of attention as both motivator and assessment marker, the goal of the project is to create a foundation of principles that will guide the design of future Computer Science badging initiatives. The team will experimentally manipulate the use of badges within an ongoing Computer Science education development project that leverages highly popular robotics competitions as the distribution channel. They will monitor and adapt the form and content of assessments and badge representations in computer science content modules to try to achieve the best possible outcomes for student participants (learner persistence, computer science content learning, and computer science career interest) as predicted by the current iteration of their badge theory. Researchers will explicitly track and publish data for underrepresented minorities, women, and economically disadvantaged students to see whether badges are particularly effective or encounter obstacles specific to these populations.

This project, conducted by the University of Pittsburgh and the University of California, Berkeley, seeks to discover what makes middle school students engaged in science, technology, engineering, and mathematics (STEM). The researchers have developed a concept known as science learning activation, including dispositions, practices, and knowledge leading to successful STEM learning and engagement. The project is intended to develop and validate a method of measuring science learning activation.

The first stage of the project involves developing the questions to measure science activation, with up to 300 8th graders participating. The second stage is a 16-month longitudinal study of approximately 500 6th and 8th graders, examining how science learning activation changes over time. The key question is what are the influencers on science activation, e.g., student background, classroom activities, and outside activities.

This project addresses important past research showing that middle school interest in STEM is predictive of actually completing a STEM degree, suggesting that experiences in middle school and even earlier may be crucial to developing interest in STEM. This research goes beyond past work to find out what are the factors leading to STEM interest in middle school.

This work helps the Education and Human Resources directorate, and the Division of Research on Learning, pursue the mission of supporting STEM education research. In particular, this project focuses on improving STEM learning, as well as broadening participation in STEM education and ultimately the STEM workforce.

The ability to express scientific ideas in both written and oral form is an important 21st century skill. Teachers, employers, and college faculty lament the inability of many high school graduates to write clearly. This deficit in writing is due in part because teachers do not have the time to provide appropriate, timely feedback to students on their writing. This project would help teachers help students achieve these skills through automating an effective feedback process, in ways that are customized to particular disciplines and local classroom needs, particularly in high needs districts. The project will contribute to knowledge about how students learn to write and how computer assisted systems can support this learning.

This project will develop and test three tools: 1) Teaching resources organized as developmental trajectories for teachers to use (e.g. from more simple to more complex; with diagnostics and strategies for addressing particular challenges); 2) A teacher dashboard that uses Artificial Intelligence tools to provide timely formative assessment to teachers by highlighting problem areas in their students' writing and peer reviews; and 3) An online teacher resource exchange to rapidly grow the set of appropriate assignments that can be used with this approach, critically filtered by student performance metrics. The project builds on a current system called SWoRD, which supports student peer reviewing in many disciplines within and beyond science. Working with six lead teachers and larger set of pilot teachers, the project will develop a trajectory of effective writing assignments in Biology, Chemistry, and Physics. In year three, there will be a summative evaluation with 90 teachers.

The University of Pittsburgh has received an NSF Improving Undergraduate STEM Education: Education and Human Resources Design and Development tier award to bring together a highly interdisciplinary team to study a suite of instructional, cognitive-skill, and social/motivational interventions that have been demonstrated to produce large improvements in learning in the context of introductory STEM courses. This research is significant because it will allow us to understand which interventions produce long-term positive outcomes, whether these interventions combine negatively or synergistically within and across courses, and the types of situations or groups of students for which they are most effective.

The project team consists of four Discipline-Based Education Research scientists (biology, chemistry, physics, and psychology), three learning scientists with expertise in social/motivational, cognitive skills or active STEM learning techniques, and a learning scientist with expertise in large-scale longitudinal data analysis with cutting edge statistical techniques. This project represents the initial phase of a new type of study termed "Intervention Science." Although interventions are relatively commonplace in educational settings, very little research has sought to comprehensively evaluate different interventions in a single, interdisciplinary methodological framework. The research plan seeks to understand the relative influence of different classroom "interventions" (e.g. "flipped" classes, active learning, peer assessment, etc.) on indicators of student success (e.g. course performance in downstream courses, participation in undergraduate research, conceptual learning gains, etc.). Construction of control and experimental groups will be accomplished using statistical techniques that assign students probabilistically to such categories. Confounding circumstances (e.g. students enrolled in different courses experiencing different interventions, or students interacting with peers in other courses) will be handled using techniques such as hierarchical linear modeling. Knowledge of the conditions in which instructional, cognitive-skill, and social/motivational interventions are most effective, and on whom, will inform the way in which faculty members implement effective interventions in the classroom and, in that way, have a significant impact on the educational experience of large numbers of STEM students nationwide.