Arthur C. Graesser - US grants

This project is being funded through the Learning and Intelligent Systems (LIS) Initiative. The long-term practical objective of the research is to develop a fully automated computer tutor. The tutor would be able to (a) extract meaning from the contributions that the student types into a keyboard and (b) formulate dialog contributions with pedagogical value and conversational appropriateness. The tutor's discourse moves include: pumping, prompting, hinting, questioning, answering, summarizing, splicing in correct information, providing immediate feedback, and rewording student contributions. The dialog contributions of the tutor would be in different formats and media: printed text, synthesized speech, simulated facial movements, graphic displays, and animation. Such an achievement will require an interdisciplinary integration of theory and empirical research from the fields of cognitive psychology, discourse processing, computational linguistics, artificial intelligence, human-computer interaction, and education. The tutoring topics will be in the domains of computer literacy and introductory medicine. Previous attempts to develop a fully automated tutor have been seriously challenged by some technical and theoretical barriers. These include (a) the problem of interpreting natural language when it is not well-formed semantically and grammatically, (b) the problem of world knowledge being immense, open-ended and incomplete, and (c) the lack of research on human tutorial dialog. Recent advances have dramatically reduced these barriers, so it is time to revisit the mission of developing an automated tutor. According to the recent research on human tutoring, a key feature of effective tutoring lies in generating discourse contributions that assist learners in actively constructing explanations, elaborations, and mental models of the material. The proposed research will advance scientific understanding of how a tutor can manage a smooth, polite dialog that promotes deep learning of the material.

The validity and reliability of answers to questions on a survey critically depend on whether the respondents understand the meaning of the questions. This project develops and tests a computer tool that assists survey designers in improving the comprehensibility of questions. The computer tool will have particular modules that diagnose each question in a survey on various levels of language, discourse, and world knowledge. For example, the critique identifies questions with low frequency words, vague or ambiguous terms, unclear relative terms, complex syntax, high working memory load, misleading presuppositions, and content that appears to be unrelated to the survey context. The computer tool will incorporate empirical findings and computational architectures in the fields of cognitive science, artificial intelligence, computational linguistics, discourse processing, and psychology. Some of these modules are so complex, technical, or subtle that they are invisible to the unassisted human eye, including experts in survey methodology, questionnaire design, and computational linguistics. This motivates the need for a computer tool to assist the research methodologist in revising questions and in learning about the complex mechanisms that underlie each component.

The computer tool will be useful to the extent that it provides an accurate and reliable diagnosis of problematic questions. The project will therefore evaluate the performance of the computer tool on several measures. Each module determines whether or not a particular question has a problem (e.g., unfamiliar technical term, working memory overload). These decisions will be compared with the decisions of experts. Other performance measures are needed because trained expert judges may miss subtle computational mechanisms. These other measures will assess whether the computer output can predict the behavior of respondents when they answer the questions: (a) behaviors of respondents that indicate they are having difficulty comprehending the question in a conversational interview (such as clarification questions of respondents) and (b) test-retest reliability of answers to questions when respondents answer a question on multiple occasions. Performance measures also will be compared for original questions, questions revised by survey methodologists who do not use the computer tool, and questions revised by survey methodologists who have had the benefit of using the tool. This research is supported by the Methodology, Measurement, and Statistics Program and a consortium of federal statistical agencies under the Research on Survey Methodology Funding Opportunity.

The Tutoring Research Group at the University of Memphis has developed a computer tutor (called AutoTutor) that simulates the discourse patterns and pedagogical strategies of unaccomplished human tutors. The typical tutor in a school system is unaccomplished in the sense that the tutor has had no training in tutoring strategies and has only introductory-to-intermediate knowledge about the topic. The development of AutoTutor was funded by an NSF grant (SBR 9720314, in the Learning and Intelligent Systems program). The discourse patterns and pedagogical strategies in AutoTutor were based on a previous project that dissected 100 hours of naturalistic tutoring sessions.

AutoTutor is currently targeted for college students in introductory computer literacy courses, who learn the fundamentals of hardware, operating systems, and the Internet. Instead of merely being an information delivery system, AutoTutor serves as a discourse prosthesis or collaborative scaffold that assists the student in actively constructing knowledge. AutoTutor presents questions and problems from a curriculum script, attempts to comprehend learner contributions that are entered by keyboard, answers student questions, formulates dialog moves that are sensitive to the learner's contributions (such as short feedback, pumps, prompts, assertions, corrections, and hints), and delivers the dialog moves with a talking head. The talking head displays emotions, produces synthesized speech with discourse-sensitive intonation, and points to entities on graphical displays. AutoTutor has seven modules: a curriculum
script, language extraction, speech act classification, latent semantic analysis (a statistical representation of domain knowledge), topic selection, dialog management, and a talking head. Evaluations of AutoTutor have shown that the tutoring system improves learning with an effect size that is comparable to typical human tutors in school systems, but not as high as accomplished human tutors and intelligent tutoring systems. The dialog moves of AutoTutor blend in the discourse context very smoothly because students cannot distinguish whether a speech act was generated by AutoTutor or a human tutor.

The proposed research will substantially expand the capabilities of AutoTutor by designing the discourse to handle more sophisticated tutoring mechanisms. These mechanisms should further enhance the active construction of knowledge. One enhancement is to get the student to articulate more knowledge, with more formal, symbolic, and precise specification; if the student doesn't say it, it is not considered covered by AutoTutor. Another enhancement is to set up the dialog so that it guides the user in manipulating a 3-dimensional microworld of a physical system; the student attempts to simulate a new state in the physical system by manipulating parameters, inputs, and formulae. The proposed research will develop AutoTutor in the domains of both computer literacy and Newtonian physics, so we will have some foundation for evaluating the generality of AutoTutor's mechanisms. AutoTutor has been designed to be generic, rather than domain-specific; an authoring tool will be developed that makes it easy for instructors to prepare new material on new topics. After the new versions of AutoTutor are completed, we will evaluate its effectiveness on learning gains, conversational smoothness, and pedagogical quality. During the course of achieving these engineering and educational objectives, the proposed project will
conduct basic research in cognitive psychology, discourse processes, computer science, and
computational linguistics. This research cuts across quadrant 2 (behavioral, cognitive, affective, and social aspects of human learning) and quadrant 3 (SMET learning in formal and informal educational settings).

This research investigates emotions during the process of learning and reasoning while college students interact with complex learning environments. College students learn about introductory computer literacy or conceptual physics on the web by an intelligent tutoring system, called AutoTutor. AutoTutor helps learners construct explanations that answer difficult questions by interacting with them in natural language and by helping them use simulation environments. AutoTutor has an animated conversational agent and a dialog management facility that attempts to comprehend the learner's contributions and to respond with appropriate dialog moves (short feedback, pumps, hints, prompts for information, assertions, answers to student questions, suggestions for actions, summaries). The emotions of the learner are monitored during this learning process by integrating state-of-the-art affect sensing technology with AutoTutor. Confusion, frustration, boredom, interest, excitement, and other learner emotions are classified on the basis of facial actions, body posture, pressure on the mouse, speech acts in dialog, mastery of the material, and the timing of interactions. One strand of research develops the affect-sensing technologies and tests their validity in classifying the learner emotions. A second line of research investigates whether learning gains and learner impressions are influenced by dialog moves of AutoTutor that are constrained by the learner's emotional state.

This research will advance education and natural language dialog technologies through a system that promotes deep learning of material in a fashion that is sensitive to the learners' emotions. A learning environment that monitors learner emotions is likely to be more motivating and personally relevant to the learner.

TRACKING MULTIMODAL COMMUNICATION IN HUMANS AND AGENTS

This project investigates multimodal communication in humans and agents, focusing on two linguistic modalities - prosody and dialog structure, which reflect major communicative events, and three non-linguistic modalities - eye gaze, facial expressions, and body posture. It aims to determine
1. which of the non-linguistic modalities align with events marked by prosody and dialogue structure, and with one another;
2. whether, and if so when, these modalities are observed by the interlocutor;
3. whether the correct use of these channels actually aids the interlocutor's comprehension.
Answers to these questions should provide a better understanding of the use of communicative resources in discourse and can subsequently aid the development of more effective animated conversational agents.

The outcomes of our observations will be modeled on controlled elicited dialog. To assure robust information on the interplay of modalities, we control the base conditions, genre, topic, and goals of unscripted dialogs. An ideal task for this is the Map Task, where dialog participants work together to reproduce on one player's map a route preprinted on the other's. The two maps, however, are slightly different, so that each player holds information important to the other. This scenario triggers a highly interactive, incremental and multimodal conversation.

In the proposed project a basic corpus of Map Task dialogues will be collected while recording spoken language, posture, facial expressions, and eye gaze. Hand gestures, discouraged by the task, will be recorded where they occur. These findings will be used in the Behavior Expression Animation Toolkit (BEAT) in order to augment the current intelligent system AutoTutor. AutoTutor has been developed for a broad range of tutoring environments that coach the student in following an expected set of descriptions or explanations. The coach-follower roles in the Map Task scenario make it possible to easily change the scenario for AutoTutor. In a series of usability experiments interactions of dialog participants with AutoTutor will be recorded. These experiments allow us to record not only the participant's impressions, but also his or her efficiency (the time to complete map, latency to find named objects, deviation of the instruction follower's drawn route from the instruction giver's model), and communicative behavior (discourse structure, gaze, facial expressions, etc.).

The research resulting from this project will benefit a large variety of fields, including cognitive science, computational linguistics, artificial intelligence, and computer science. In addition, the integration of the modalities into a working model will advance the development and use of intelligent conversational systems.

This award focuses on examining the effectiveness of using animated pedagogical agents (APAs) as external regulatory agents designed to foster middle school and college students' understanding of complex and challenging science topics (e.g., the circulatory system). Contemporary cognitive and educational research provides evidence that the potential of computer-based learning environments for facilitating learning may be severely undermined by students' inability to regulate several aspects of the learning. For example, students should regulate key cognitive, metacognitive, motivational, social, and affective processes in order to learn about complex and challenging science topics. This research will be conducted in the context of a mixed-initiative intelligent tutoring system called AutoTutor that simulates the discourse patterns and pedagogical strategies of human tutors. The focus of our grant is on conducting interdisciplinary research examining: (1) the role of embedded animated pedagogical agents in collecting data of the complex interactions between cognitive and metacognitive processes during learning about complex science topics with AutoTutor; (2) the effectiveness of animated pedagogical agents as external regulating agents used to detect, trace, model, and foster students' self-regulatory processes during learning about complex science topics with AutoTutor; and (3) the effectiveness of scaffolding methods delivered by animated pedagogical agents in facilitating middle school and college students' selfregulated learning about complex science topics with AutoTutor.

This is funding to support travel by 20 students currently enrolled in PhD programs in the United States to participate in the AI-ED Doctoral Student Consortium, at the upcoming International Artificial Intelligence in Education Conference (AI-ED), to be held July 9-13, 2007, in Los Angeles, California. The AI-ED International Conference is the premier biennial event for promoting promotes rigorous research and development of interactive and adaptive learning environments for learners of all ages; AI ED will be the 6th event in the series. The interdisciplinary areas that AI-ED represents, comprising cognitive science, computer science, and educational technology, are critical research domains that enhance the effectiveness and usability of software learning systems. Active participation of young researchers in this conference is very important, both for the health of the field and for the researchers themselves. The AI-ED '07 Doctoral Consortium provides a unique opportunity for PhD students partway through their dissertation research to receive valuable feedback and individual mentoring from top researchers in the field. This support for doctoral students is particularly apt given that currently only 4% of the over 600 members of the AIED society are students, even with 44 countries represented. It is thus timely for NSF to offer this student support. The PI and co-PI both hold active leadership roles within AI-ED '07, and both hold over $2 million of NSF grants in the areas of human-language and advanced learning technologies. The AI-ED '07 Doctoral Consortium Committee will be comprised of the two PIs together with the current president of the AIED society.

Broader Impact: Bringing young and creative researchers to AI-ED '07 will help advance an important and socially valuable interdisciplinary research field. For many graduate students, the cost of attending the AI-ED conference exceeds their travel budget. Thus, NSF funding will significantly impact the careers of the next generation of AIED researchers, by enabling a number of them to take part in an important event they would otherwise have to miss; in particular, those who lack funding from other sources (e.g., advisor's grants). The students will have an opportunity to gain wider exposure in the community for their innovative work, and to obtain feedback and guidance from senior members of the research community. Participation will also help foster a sense of community among these young researchers, by allowing them to create a social network both among themselves and with senior researchers at a critical stage in their professional development. The PI and co-PI have indicated that they will act to assure participation by members of traditionally under-represented institutions, and will pay close attention to inclusion of minorities and women.

The Workshop on the Question Generation Shared Task and Evaluation Challenge aims at establishing a community consensus with respect to various issues related to offering a shared task on Question Generation, including a clear definition of the main task and subtasks, data collection and annotation processes, data annotation schema, type of evaluation and evaluation metrics. The Natural Language Generation community has currently identified shared tasks as a potential venue to provide a focus of research in the field and increase the visibility of Natural Language Generation research in the wider Natural Language Processing/Computational Linguistics community. This award contributes to this effort by defining a novel shared task, the Question Generation task. The workshop organization is guided by the Desiderata for Evaluation of Natural Language Generation outlined at the recent NSF/SIGGEN Workshop on Shared Tasks and Comparative Evaluation in NLG held in 2007. The goal of a representative group of researchers from various areas of research (Natural Language Generation, Intelligent Tutoring Systems, Artificial Intelligence in Education) participating in the workshop discussions is to produce a to-do list for preparing and offering a Question Generation task at Generation Challenges 2010. A post-workshop report will outline the discussions and decisions of the workshop and will be made available to the research community. Due to the importance of Question Generation in learning technologies such as Intelligent Tutoring Systems, the workshop will have broader impact on improving the teaching and learning of STEM disciplines through improved educational software applications.

The ways people use words can provide insights into their thoughts, motives, and relationships with others. Discourse patterns can be modeled to predict deception, status, group processes, emotional states, and personality. An important question is whether the relationship between discourse and social dynamics can be extended to create predictive models of social processes across multiple languages and cultures, and among military, political and civilian organizations in authoritarian regimes. The specific goals of the project are a) to define and compare the ways natural language reflects social dynamics through the analysis of a wide range of documents across languages and cultures, b) to develop annotated and socially indexed multi-language databases of communication (e.g., English, Arabic, Chinese, Spanish, and Korean), along with preliminary research tools that can enhance our ability to conduct research across languages and cultures, and c) to address key national security questions emerging from the collection of vast amounts of digitized documents and communication, such as the capability to understand past actions and cognitions of previous regimes and assess emerging threats.

The project will focus on three types of social dynamics: a) Leadership, identity and group dynamics, and the degree to which the psychological and social identities of people and groups can be determined through writing and speech; b) Cohesion of text and social processes, and whether it is possible to identify the connections among people in a social group, such as the stability of the group and likelihood of defection, by comparing the ways the group members separately use words; and c) Deception and misinformation, and whether it is possible to identify deception and deceptive intent from language-based cues. A cross-cutting research theme is the complexities of conducting computational analysis of discourse and social dynamics across multiple languages and translated corpora. For example, to what degree does translation distort the link between language and social dynamics?

The project will advance social science theory, developing new models linking language use with social dynamics and providing a new way of looking at large cross-sections of a society. New computational methods will be developed that can be applied to large, representative samples of text from across an entire country's political and cultural landscape. These methods, unlike traditional analytical methods, do not require individual analysis by highly trained social scientists and historians and can provide the capacity to process documents that may number in the millions. These developments and findings will advance analyses of social dynamics that were not previously possible and can assist scientists and practitioners, such as security analysts, in understanding information flow and social structure at a macroscopic level. Such knowledge could potentially help security analysts to pinpoint individuals and groups that merit direct study by trained experts.

This is funding to support travel by 18 intermediate and advanced doctoral students to participate in the AI-ED Doctoral Student Consortium, at the upcoming International Artificial Intelligence in Education Conference (AI-ED), to be held to be held in Brighton, United Kingdom from July 6 to 10, 2009. The AI-ED International Conference is the premier biennial event for promoting promotes rigorous research and development of interactive and adaptive learning environments for learners of all ages; AI ED will be the 7th event in the series. The interdisciplinary areas that AI-ED represents, comprising cognitive science, computer science, and educational technology, are critical research domains that enhance the effectiveness and usability of software learning systems. Active participation of young researchers in this conference is very important, both for the health of the field and for the researchers themselves. The AI-ED Doctoral Consortium provides a unique opportunity for PhD students partway through their dissertation research to receive valuable feedback and individual mentoring from top researchers in the field. The PI and co-PI both hold active leadership roles within AI-ED, and both hold over $2 million of NSF grants in the areas of human-language and advanced learning technologies. The AI-ED Doctoral Consortium Committee will be comprised of the two PIs together with two other senior professors/researchers.

Broader Impact: Bringing young and creative researchers to AI-ED will help advance an important and socially valuable interdisciplinary research field. For many graduate students, the cost of attending the AI-ED conference exceeds their travel budget. Thus, NSF funding will significantly impact the careers of the next generation of AIED researchers, by enabling a number of them to take part in an important event they would otherwise have to miss; in particular, those who lack funding from other sources (e.g., advisor's grants). The students will have an opportunity to gain wider exposure in the community for their innovative work, and to obtain feedback and guidance from senior members of the research community. Participation will also help foster a sense of community among these young researchers, by allowing them to create a social network both among themselves and with senior researchers at a critical stage in their professional development. The PI and co-PI have indicated that they will act to assure participation by members of traditionally under-represented institutions, and will pay close attention to inclusion of minorities and women.

This research will explore interactions between cognition and emotion during the learning of scientific methods in the context of a computer tutoring environment. The primary focus will be on the relations between impasses, cognitive disequilibrium, and the affective-cognitive state of confusion. Confusion correlates with learning gains because it is diagnostic of cognitive disequilibrium, a state that occurs when learners face obstacles to goals, contradictions, incongruities, anomalies, conflicts, and system breakdowns. Cognitive equilibrium is normally restored after thought, reflection, problem solving and other effortful cognitive activities. Therefore, pedagogical tactics that challenge, perplex, and productively confuse learners are stimulating alternatives to the typical information delivery systems in education that promote shallow knowledge in the comfort zone of the learner, but rarely deep comprehension. This research will develop tutorial interventions that induce, track, and regulate confusion and cognitive disequilibrium in the minds of learners, as well as the cognitive and emotional mechanisms that restore cognitive equilibrium. The research has three specific objectives: (1) To promote deep learning by developing tutorial interventions that experimentally induce impasses, cognitive disequilibrium, and the resulting confusion; (2) to integrate sensing devices and signal processing algorithms that detect and track the associated confusion; and (3) to develop affect-sensitive pedagogical strategies to help learners regulate their confusion. The three objectives will be accomplished by augmenting an existing Intelligent Tutoring System (ARIES, Acquiring Research Investigative and Evaluative Skills) with technologies that automate assessment of emotion and cognition, as well as an intelligent handling of emotions. State-of-the-art sensing devices detect relevant emotions during learning (confusion, frustration, boredom, flow/engagement, delight, surprise) on the basis of the dialogue history, facial expressions, and body posture. The ARIES system promotes scientific inquiry skills by presenting case studies that exhibit flawed scientific methods and that require learners to offer thoughtful critiques on the scientific merits of the studies. The critiques encourage (a) the general cognitive processes of drawing inferences, constructing causal models, identifying problems, and asking diagnostic questions and (b) skills that directly target scientific reasoning, such as stating hypotheses, identifying dependent and independent variables, isolating potential confounds in designs, interpreting trends in data, and determining whether data support predictions. Students interact with ARIES through conversational trialogues in natural language with two animated agents: a tutor agent and a peer agent. Cognitive disequilibrium is created when the agents produce messages with contradictions, conflicts, and clashes with what the student knows. Correct information eventually emerges in the trialogue, which restores cognitive equilibrium.

The broader significance and importance of the project is to advance science education, intelligent learning environments, and human-computer interfaces. It is widely acknowledged that the level of science understanding among students and adults in the United States needs improvement and does not compare favorably with several other nations. The proposed research will help fill this gap by developing technological interventions to fortify citizens and aspiring scientists with the skills needed for critical thinking, complex reasoning, and problem solving in science. The project will develop intelligent learning environments targeted for deeper learning, which is needed for a technologically sophisticated workforce, and for a motivating learning experience, which is expected in recent generations of students. The project will develop advanced sensing devices for detecting emotions and cognition, a contribution that should impact the fields of human-computer interaction, cognitive science, and the learning sciences.

This award supports a student session at the 2nd Workshop on Question Generation. The aim of the workshop is to strengthen the Question Generation (QG) research community and create consensus with respect to QG as a shared task. The 2nd Workshop on Question Generation is a 1-day workshop organized into two sets of sessions. In the morning sessions, regular paper presentations on general topics related to QG are scheduled. The afternoon sessions are dedicated to discussions and presentations related to QG in Intelligent Tutoring Systems, one category of shared tasks identified at the previous Workshop on The Question Generation Shared Task and Evaluation Challenge. As part of the afternoon sessions we have a student session. Attending the workshop will greatly impact the scientific awareness and skills of students. It will allow students to become real contributors and leaders of the QG research community by engaging in discussions at the workshop about important research issues in this area of research. The workshop is a great venue for student to be exposed to interdisciplinary research which will definitely strengthen their skills. The broad implications of QG research will increase students? motivation to do research in this area. The students attending the workshop will be among future leading scientists who could boost our national STEM education, a mission of particular interest to the National Science Foundation, due to the importance of QG in learning technologies. We will encourage participation of groups underrepresented in science and engineering, including minorities, women, and persons with disabilities.

The core research question of this proposal is how interactions between learners (i.e., individual differences), instructional materials (i.e., the text), and learning activities (i.e., the task) modulate engagement during learning of critical thinking skills and scientific reasoning. The proposed research will address this goal by: (a) systematically investigating the mechanisms that facilitate or hinder engagement, and (b) leveraging these insights towards the development of interventions that promote persistent and productive engagement trajectories during deep learning. The work will be conducted at the University of Memphis. The research subjects will be undergraduate students.

The research design includes four experiments in which learning gains and self-reported engagement, physiological arousal, eye gaze patterns, and facial features will be tracked while learners study instructional texts. Comprehending these texts for mastery requires active engagement as learners generate inferences, understand causality, identify problems, discriminate the quality of experimental designs, and ask diagnostic questions. Analyses will be conducted using nonlinear time series analysis techniques, such as recurrence quantification analysis. The project evaluation will include an expert advisory committee that will be used to critically review the investigators' findings and interpretations. In addition, the investigators will develop and validate a web-based computer program that dynamically tailors both the instructional text and the learning activity to the needs and learning styles of individual learners to enhance engagement. The proposed research will balance the theoretical goal of theory building and model testing via systematic experimentation with the practical goal of developing innovative advanced learning technologies that aspire to promote engagement and learning of difficult subject matter.

This research is important in the STEM education field's ongoing efforts to increase engagement and the productivity of learning of STEM subject matter. If the research is successful, the derivative knowledge and tools will be significant contributions and could be applied widely in other intelligent tutoring systems, other instructional technologies, and in our understanding of student learning in general. This research is potentially transformative in two ways. First, it will provide a detailed understanding in real time of engagement at the micro, relational level of student, task, and materials. Second, the creation of an intelligent tutoring system based on these findings holds the possibility of being able to 'correct' low levels of student engagement on a moment-to-moment basis and therefore boost learning productivity while increasing student satisfaction and engagement with the experience. Dissemination will include the public availability of the technological tools as well as contributions to the scholarly literature.