Justine Cassell - US grants

The relationship between speech and gesture in narrative discourse will be investigated by animating the interaction between the two media on a three-dimensional interactive animated model -- a graphic figure on a computer screen that would ultimately be able to "communicate" with a listener, gesturing appropriately while producing speech. Research demonstrates that gesture is intimately linked to speech both in the production and the comprehension of narrative discourse. This interaction exists at many different levels of organization. The relationship between speech and gesture can be specified at the level of the timing of gestures with respect to the prosodic contour of a clause. It can be specified at the level of semantic and pragmatic descriptions of an utterance, and at the level of narrative structure with respect to the types of gestures found at different points in a narrative. Interactive activities at the host institution include: meetings once every two weeks, open to graduate students and junior faculty, to discuss particular problems in accomplishing work and constructing a supportive environment for achieving professional goals; hour-long meetings the alternate weeks of the month with undergraduate students, to discuss similar topics; sponsorship of two visiting cognitive scientists per semester, where these will be women who have made 'non-standard' career choices and/or have written about professional issues facing women in academia; co- teaching a seminar in Computational Linguistics with Ellen Prince on discourse processing and gesture. All of the parts will be carried out under the aegis of the Computer Science Department, but will be open to other members of the Cognitive Science community at the University of Pennsylvania. This project furthers VPW program objectives which are (1) to provide opportunities for women to advance their careers in engineering and in the disciplines of science supported by NSF and (2) to encourage women to pursue careers in science and engineering by providing greater visibility for women scientists and engineers employed in industry, government, and academic institutions. By encouraging the participation of women in science, it is a valuable investment in the Nation's future scientific vitality.

While multiple, and technological literacies have become the topic of considerable research, reading and writing literacy remain the basis of education, and the prerequisites to science, mathematics, and technology fluency. Reading and writing do not start in school, however. Children prepare themselves for later literacy long before first grade. And preparation for literacy consists of more than becoming aware of text. Children learn to treat language as an object (metalinguistic awareness), and to create and maintain cohesive text (decontextualized language) first in oral language. Many of these emergent literacy skills are acquired in language play and in storytelling among peers. In general, however, very few technologies are available for supporting children's storytelling and story writing in sociocultural context for later literacy, despite its importance in children's cognitive, communicative, and linguistic development as a whole.

This project lays out a program of research designed to address a specific need of young children -- to learn how to write -- based on one specific ability of young children -- the ability to tell stories. Current views of the relationship between oral and written literacy hold that the development of children's written literacy is intertwined with the development of their oral literacy skills. We believe that technology, and particularly tangible non-screen-and-keyboard based technology, can play a unique role in supporting the emergence of writing literacy by building on one particular aspect of oral literacy, children's story telling. We propose that Story Listening Systems that actually listen to children's stories and interact with them about the stories not only support the development of emergent oral literacy skills but also foster writing skills and the transition from oral to written literacy. The model behind the Story Listening System embodies four essential traits that enable it to effectively scaffold written literacy. The four traits are to (a) depend on children's oral storytelling skills to bootstrap literacy, (b) introduce peers as playmates in the system or with the system, (c) invite the kind of embodied play away from the desktop that is most comfortable for young children, (d) allow children to construct their own personally meaningful content.

In order to evaluate the effectiveness of these systems, children between the ages of 4 and 7 will engage with Story Listening Systems and we will assess changes in three crucial predictors of written literacy: (i) use of decontextualized language (language removed from its original context, and reworked for a new audience), (ii) metalinguistic awareness, (iii) collaboration with peers to make meaning. We will also evaluate an additional implication of the SLS: that support for emergent literacy can be made culture-inclusive and therefore can be used to invite more children into emergent writing literacy because the language forms with which they are familiar are embedded in the system.

The outcome of the proposed work will be four-fold: (i) a suite of story listening literacy technologies, (ii) a substantive body of data evaluating the role of these technologies in children's writing, (iii) a fuller understanding of the general mechanisms underlying children's development of literacy skills, (iv) a generalized set of design principles that link features of technology with features of children's written literacy acquisition. The design principles will be for two audiences: the design community creating children's digital technologies (e.g. everyone from mainstream toy industries making physical artifacts to game designers to traditional educational/learning software designers) and those actively studying and teaching written literacy (e.g. developmental psychologists, educators). This work has implications, then, for the communities of academics, educators, technologists, and policy organizations concerned with placing technology in school contexts.

This award is for funding to subsidize travel, conference, and housing expenses of students selected to participate in the ACL Student Research Workshop to be held during the joint ACL on July 21st to 26th, 2004 in Barcelona, Spain. The Association for Computational Linguistics (ACL) is the primary international organization in the field of natural language processing and language engineering. The Association's annual conference, which rotates among the Americas, Pacific-rim countries, and Europe/Africa, is the major international meeting in the field. Computational linguistics is a multidisciplinary field of increasing importance to science, commerce, and society. This workshop contributes to the professional development of young scientists who will lead this growing field in the coming decades.

The workshop format allows students to present their research and receive feedback from a panel of established researchers in the field. It provides students with invaluable exposure to outside perspectives on their work at a critical time in their professional development through feedback from the panel and other student participants. Also, student participants will be able to see each other's presentations and feedback, which could lead to a general raising of standards and expectations, strengthening future student research. The ACL Student Workshop is an inexpensive yet highly effective means of encouraging young and upcoming computational linguists. The intimate format encourages the student participants to begin building a rapport with established researchers. The workshop should contribute to the maintenance and development of a skilled and diverse natural language processing workforce, helping to produce a pool of researchers with the scientific and engineering knowledge required for future applications of natural language processing. It should help to foster international scientific exchange, providing merit-based opportunities for future leaders of the field to attend a conference that might otherwise be less geographically diverse than is desirable for the field. In addition, by building a supportive environment for these students, it is more likely that down the road, they will lend a supporting hand to other students who follow.

From mathematics word problems to great literature to science texts, being able to read and write is of paramount importance to academic success. In addition, with the Federal mandate of "no child left behind" an increasing national priority is learning objectives that prepare students to do well on national standardized tests, and many of these learning objectives rely on literacy skills. Yet literacy remains a critical unsolved issue in our educational system, with national test scores showing that non-white children (e.g., African Americans and Hispanics) still score well below their Caucasian counterparts. Increasingly, approaches to this problem derive from the recognition that primary school education is based on a set of mainstream oral practices and literacy-preparation skills. Because many children do not share the same cultural experiences typical of mainstream culture, educational practitioners have applied a cultural-historical approach to identify common characteristics of ethnic groups, and then designed cultural supports for literacy learning by children of diverse backgrounds. But while classroom practice has been influenced by this approach, and training programs for parents have been instituted along these lines, it has been rare to see technological learning environments that leverage diversity in any but the most superficial ways (for example identical content characters with different skin color, or content based on traditional associations such as the Anancy myths). The PI believes she can do much better. To this end, in the current project she will first conduct an in-depth investigation into African-American peer-oriented oral language and literacy practices, in order to understand how cultural practices support individual learning and development in this population. Using these findings, she will then design a virtual peer that engages in authentic cultural practices with African-American children as a bridge to school-based literacy. As the technology is developed, the PI will bring it into classrooms and community centers, formatively evaluating and iteratively redesigning as necessary until the virtual peer is capable of being like African-American children in important ways, can exploit that affinity to establish rapport, and can leverage this rapport and linguistic interaction to help the child acquire literacy skills. Project outcomes will include a rich comparative description of the language and nonverbal practices of African-American and Caucasian children in peer-emergent literacy interaction, a set of behaviors that allow a virtual human to establish rapport with users (including local-level moment-by-moment interactional behaviors and larger scale culturally specific practices), and a Flash virtual peer and set of virtual peer behaviors that have been shown to improve literacy in the children interacting with the virtual peer, both for Standard American English (SAE) speakers and African American Vernacular English (AAVE) speakers. A summative evaluation will address transfer from interaction with the virtual peer to other standard literacy contexts and tests.

Broader Impacts: This project addresses fundamental issues relating to acquisition of literacy skills by young children who are members of an underserved population, and will lead to new technology for improving their literacy readiness. The PI will pursue an innovative program of dissemination of results and research practices that involves undergraduates from HBCUs, local schools with high populations of African Americans, and local churches and community centers, in order to broaden in the short term participation by African-American undergraduates in the engineering research behind this project and in the longer term participation in STEM by young African-American school children.

With the advent of increasing numbers of increasingly smart machines, there is a growing need to develop technologies that are not only smart, but sensitive to the people and the other machines around them, and sensitive to the context in which they are used. Such an understanding will permit the development of technologies that can coordinate their interactions with humans in a more natural, seamless and fluid fashion. To meet these goals, this research program focuses on three critical yet under-studied contexts of interaction, each of which represents a different constraint upon interpersonal communication: (1) the physical context of shared visual access, (2) the social context of rapport, and (3) the biological context of aging. While some research has been conducted on each of these contextual factors, none has addressed their interaction, nor gathered them into one broader conception of the role of context in interpersonal coordination. This research applies a theory-driven design approach that includes experimental studies, theory development, computational modeling, system implementation and evaluation. In particular, the research program proposes: a) A rigorous study of human-to-human communication using elicitation experiments to develop a more detailed understanding of interpersonal communication across a range of contexts; b) A formalization of the findings into computationally explicit forms that provide predictions of behavior and capture the observed behavioral patterns; c) Integration of the models into a dialogue manager that is implemented within a larger computational architecture; and, d) Evaluation of the implemented system by having untrained humans interact with the system in such a way as to evaluate its effectiveness and reveal gaps in the underlying models as well as in our theoretical understanding.

The outcome of this research will advance our theoretical understanding of the role various contextual factors play during interpersonal communication. The results will be useful to a variety of scientific communities including those that study basic human communication (e.g., psychologists, linguists and communication researchers) and those that study interactive computational systems (e.g., computer scientists, computational linguists, and interaction designers). The research will also provide practical design guidelines and a general computational model that describes how machines can make intelligent choices on the basis of these contextual factors during everyday interactions. At a practical level, the general computational model can be applied by technologists developing many different technologies, such as embodied agents, large-scale displays, ubiquitous computing, in-car navigation, and assistive technologies for the elderly and those with cognitive impairments.

This project will implement and evaluate technological supports for African-American Vernacular English(AAVE)-speaking children to learn Standard American English while engaging in problem-based scientific inquiry. The technology consists of virtual peers that collaborate with children to solve a bridge-building problem, while scaffolding the notion that different kinds of language are appropriate for different conversational contexts. The work relies on the recognition that primary school education is based on a set of mainstream oral practices and literacy-preparation skills, and yet all children do not share the same cultural experiences typical of mainstream culture, nor come to school speaking the same dialect of English. Similarly, while traditional science classrooms have emphasized a particular style of scientific discourse, not all children come to school with the mastery of these discourse styles. Scientific inquiry is at the heart of the contemporary science classroom but it is usually defined according to a specific cultural tradition that privileges individual opinion, 'talking back' to the teacher, and criticism of others; a tradition that may not be shared by all students, and which may have ramifications for science achievement among diverse populations. A unique approach for integrating cultural authenticity into learning technology will be pursued: (1) carrying out an in-depth investigation into AAVE peer-oriented language and nonverbal communicative behaviors. The corpus of data obtained from this study will be shared with all interested researchers via the Penn Linguistic Data Consortium; (2) Two technological innovations will extend prior work on virtual peers so as to make possible the current work: (a) PIPER, a new platform for rapid prototyping and implementation of virtual peers so that each of the virtual peers does not require extensive re-implementation as it did in our Flash days; (b) AVP, an authoring system for virtual peers so that children themselves can program the virtual peer as a way of actively engaging with the technology, with code-switching and with collaborative science inquiry and then description of that inquiry to a teacher; (3) evaluating the technologies with respect to their role in improving children?s use of SAE, their educational self-efficacy, and their learning gains in second grade standardized science measures.

The broader significance and important of the work lies in: (1) the potential to substantially increase access to reading, writing, and science literacy for under-served, at-risk children, and to thereby decrease the Black-White achievement gap; (2) technological innovation that will allow other researchers to quickly prototype and implement virtual peers and pedagogical agents that speak different dialects and language, and that can be programmed by their designers, and by their users; (3) an innovative program of dissemination of results and research practices that involves publication and presentation of results, sharing the corpus of data via the Penn Linguistic Data Consortium, but also the involvement of local schools with high populations of African-Americans, local churches and community centers, and informal education institutions such as science centers and children's museums.

This is funding to support a pair of events in conjunction with the 2008 Interaction Design for Children conference (IDC08), which will take place June 11-13 in Evanston, Illinois: a Doctoral Consortium which will bring together approximately 9 promising doctoral students from the United States and abroad along with 3 distinguished research faculty; and a pre-conference Workshop on Designing Technologies for Children with Special Needs that will culminate in a keynote lecture that will explore the challenges, opportunities and complexities behind producing international versions of a children's television program. The annual IDC conference is the leading international forum for the presentation and discussion of interaction design for children, from the perspective of research as well as practice. IDC08 will strive to expand the reach of previous conferences, both in terms of the number and kind of attendees, and in terms of the number and kind of children that we target with our technologies. The goals of the Doctoral Consortium are to increase the exposure and visibility of the participants' work within the community, to help establish a sense of community among this next generation of researchers, and to help foster their research efforts by providing substantive feedback and guidance from a group of senior researchers in the area in a supportive and interactive environment. Student participants in the Doctoral Consortium will make formal presentations of their work and will receive feedback from a faculty panel; the feedback is geared to helping students understand and articulate how their work is positioned relative to other research, whether their topics are adequately focused for thesis research projects, whether their methods are correctly chosen and applied, and whether their results are appropriately analyzed and presented. The Workshop on Designing Technologies for Children with Special Needs will allow intellectual exchange among top researchers and designers in this field. Discussions about challenges for working with special populations, new methods for approaching these challenges, and new directions for research and design will contribute to and expand the research perspective of all participants. During the main conference attendees of both events will present their work in poster sessions, and abstracts will be included in the IDC Conference Proceedings as well in the ACM Digital Library.

Broader Impacts: The pre-conference workshops at IDC08 will bring together some of the best students, researchers and practitioners in the field of interaction design for children, and will thereby afford the younger participants a unique opportunity to gain wider exposure in the community for their innovative ideas while also allowing them to create a social network both among themselves and with senior colleagues in the field. Since the IDC conferences host a diverse group along several dimensions (such as nationality, scientific discipline, and research specialization), participants' horizons will be broadened and new collaborations will emerge, to the future benefit of the field.

Developed around the research theme of computer innovation in modeling and simulation of complex human and physical systems, co-PIs and their students will work with local classroom teachers with the goal of taking cutting-edge modeling and simulation from the fields of computer science, physics and astronomy and translate them into tools that can teach grade-level appropriate science content. Each graduate fellow will be partnered with a local classroom teacher and will engage in a training, development, and implementation process over the entire year. External evaluators will help us assess progress towards the goals and benchmarks of the program.

More broadly, this work will build the graduate fellow?s teaching, analytical and communication skills. The fellows will gain a deeper understanding of the foundations of their research in deconstructing and isolating the basic concepts of computer science, physics, and astronomy from their work to teach them to the classroom students. Classroom teachers will be exposed to cutting-edge scientific research and the research process as well as forming lasting partnerships with university researchers and scientists. Classroom students, many from traditionally underrepresented populations, will gain an improved understanding of science content and the research process, enabling them to be more informed consumers of science content presented in the media. Classroom students will also be encouraged to pursue careers in science, technology, mathematics and engineering from interactions with the fellows as role models and mentors.

This project is supporting the first in-person meeting of the Global Learning Council (GLC). Started in 2013, the GLC includes members from industry, academia, non-profits, and government. At this conference they will discuss technology-enhanced learning, focused on the application of learning science to the development of learning tools, data on learning gains, and the science of how students learn.

Specific conference sessions include a) Best Practices, b) Big Data, and c) K-16. Working groups will focus on additional topics such as a) funding for educational research in the era of online courses and other technology-enhanced learning tools, and b) policy challenges related to scaling research-driven outcome-based learning. The conference will lead to several deliverables which include case studies that demonstrate the importance of data in improving technology enhanced learning, recommendations on the next steps and processes in using student outcome data across different sectors (such as educational, workforce, company, and philanthropic), announcement of a new collection of learning outcome data donated by a number of the institutions represented in the GLC, the announcement of a set of tools that make learning analytics accessible to an increasing number of the stakeholders involved in the development and dissemination of effective teaching. These and other deliverables will be made available through the Global Learning Council site and the site for the Simons Initiative at Carnegie Mellon. Conference attendees will provide evaluation feedback. This project is funded by the Division of Undergraduate Education through the IUSE program.

The Cyberlearning and Future Learning Technologies Program funds efforts that will help envision the next generation of learning technologies and advance what we know about how people learn in technology-rich environments. Cyberlearning Exploration (EXP) Projects explore the viability of new kinds of learning technologies by designing and building new kinds of learning technologies and studying their possibilities for fostering learning and challenges to using them effectively. This project seeks to understand, and capitalize on, how teachers or tutors build rapport with learners by building technologies that support rapport. Research will study what rapport with learners looks like, when students deploy rapport techniques, when and how deploying rapport techniques (whether by people or automated agents) increases learning, and how rapport evolves over time. The project will build software that can help measure rapport between learners and with computers.

The project will begin by building a multimodal sensing rapport-detection system , based on recent advances in computer vision, signal processing, and machine learning which will automatically recognize audio and visual behaviors during learner interaction with an intelligent tutoring system. Human-human tutoring interactions will be used to guide development of the rapport detection system. Both short term and longitudinal analyses will be conducted using students working with an AI-based math tutor, focused on their visual behaviors (head gaze estimation will be used to measure facial action units and gestures like head nods or shakes, and mutual gaze between humans), verbal behaviors (using CoreNLP and other software to detect verbal utterances that represent rapport-related social constructs such as politeness, friendship, etc.), and entrainment behaviors (synchrony or asynchrony, divergence and convergence). The project will then design RAPT, the Rapport-Aligned Peer Tutor, which encompasses both the rapport detection system and an intelligent pedagogical agent that accounts for the persistent social states of rapport and non-rapport. Mockups/simulations of the interface will be used to test the designs before the full pedagogical agent is built. Trials will be conducted in 9-11 grade classrooms working with an intelligent geometry tutor using a two-iteration design-based research study.