David McNeill - US grants

ABSTRACT By the time children are five years old, they are using many of the same linguistic and gestural devices that adults do to mark the relationship between events in a story. This fact has lead many researchers on narrative skills to presuppose the notion of event. The present study asks whether appearances may not be deceiving; whether children and adult's similar linguistic means function in fundamentally different ways. In the first phase of the study adults and children aged 4-5, 7-8, and 10-11 will watch a short cartoon and then narrate the story to a same-age listener. Analyses of verbal and gestural transcripts will focus on the linguistic devices for conveying temporal structure in narrative, and on the use of non- representational gestures which adults produce when they are introducing and commenting on narrative events. In the second phase, the same subjects will be presented with stills from the stimulus cartoon and asked to "group the pictures that belong together". Event groupings will be compared with a frame- by-frame analysis of the stimulus cartoon, and with each subject's choice of temporal conjunctions and placement of beats and deictic gestures. In this way, this study begins to find out how children convey the interrelationship of events in narrative with the linguistic and gestural means at their disposal, and contributes to an understanding of the interaction between cognitive ability and linguistic skills in the development of narrative competence.

The symptomatology of aphasia after left-hemisphere damage should not distract us from the effects of right-hemisphere damage on the verbal performances of individuals who suffer it. These effects are subtle and are not manifested in a single subsystem like language, but affect the patient's ability to represent complex structures such as narratives and other forms of discourse. By considering complex performances (narratives, living-space descriptions, and descriptions of networks), plus tracking the nonverbal components of the performance (the speaker's gestures that accompany speech), we can study the deficits due to right-hemisphere damage in a clear and rigorous way. Our hypothesis is that normal speaking -- with a dual presentation of verbal and imagistic forms of representations - - involves the cooperative interaction of both hemispheres, each making its own specific contribution. Our hypothesis recognizes that speech and gesture constitute, at some level, a single system. Spatial representations thus can condition linguistic choices and vice versa. Patients with right hemisphere damage syndrome have seemingly unimpaired speech, but manifest poor story structure and gesture. There can be dissociation such that the strictly linguistic contributions to discourse structure occur in the absence of the appropriate imagery. This produces an 'empty' linguistic performance, not backed by the visuospatial representations that are part of normal metanarrative structure. In contrast, in left hemisphere patients, there are intact visuospatial representations, however imperfectly they may be expressed in linguistic terms, and these show themselves in both gesture and speech. Thus, according to our hypothesis, the narrative performance of left hemisphere patients may be underestimated by a too-narrow focus on the linguistic details of narrative speech and the performance of right hemisphere patients, due to the same narrow focus, may be overestimated . The relevance of this research for health lies in the understanding it will offer the contribution of the right and left hemispheres to language, gesture and discourse; of the role it tests of spatial imagistic representations in the formation of narrative and other discourse beyond specific linguistic choices; and of possible new approaches to therapy and evaluation of hemisphere damaged patients.

Behaviorally Situated Avatars for Tutoring: Beyond the Talking Head and Animated Icon
Abstract

One-on-one tutoring is a critical component of teaching and learning. This project investigates technology to deploy an anywhere-tutor anywhere-student system that permits a tutor to provide instruction via an animated avatar. Exploiting psycholinguistic information, the system translates time-situated pointing and communicative gestures performed on a LCD tablet in conjunction with speech into a stream of behaviorally correct, spatially situated 3D gestures performed by the avatar tutor. A camera tracks the student, facilitating socially aware behavior by the avatar. The system exploits the student's ability to use body motion cues in the embodied avatar to direct her attention to appropriate locations of projected graphics that serve as artifacts of instruction. The technology will potentially facilitate a tutoring program in which students may connect to a pool of tutors for help with a variety of topics. The approach essentially implements tutoring 'telephone handsets' at both ends of the interaction. Tutors may be distributed across multiple sites at their convenience and take calls as they are needed. Widespread access to such tutoring will facilitate the vision of universally available educational opportunities. The project will investigate the necessary artificial intelligence and interaction technologies and evaluate the efficacy of this tutoring methodology.

When we speak, our embodied behavior of gesture, gaze, posture, and facial expression are brought into the service of the communicative process. This research is grounded in psycholinguistic theories of multimodal human communication. One path from multimodal behavior to language is bridged by the underlying mental imagery. This spatial imagery, for a speaker, relates not to the elements of syntax, but to the units of thought that drive the expression (vocal utterance and visible display). Gestures reveal the focal points of the accompanying utterances, and relate to the meaning of the newsworthy elements of the unfolding discourse. This is the underlying premise of the current project, in which the PI will focus on math discourse and education for blind students.

Mathematical reasoning is rich in spatial imagery that is revealed in gesture. Furthermore, gesture has the capacity to create images of the math concepts that serve as "objects of contemplation." When a graphic/illustration is available for math instruction, the discourse stream is typically situated with gestures of spatial reference into the graphic. Research with individuals who are blind suggests that they have remarkable capacity for visual imagery, memory, and conceptualization. However, students who are blind tend to lag behind sighted students in mathematics education. The PI posits that a significant impediment to math instruction for students who are blind lies in the lack of visual access to the embodiment of the instructor. He notes that we have, in such students, a population that is able to access graphical content (through tactile image technology), but is not visually aware of the embodied behavior of the interlocutor. He proposes to remedy this problem by affording blind students a sense of embodiment awareness, with the help of augmentative communication approaches that employ tactile devices to provide elements of embodiment awareness. To this end, the PI will perform a series of "perception and action" experiments to assess the efficacy of such devices, followed by a second series of experiments with blind and sighted students in mathematics instruction, to be captured on video, and with pre- and post- tests, to assess the quality and quantity of imagistic content both conveyed by the teacher and by the student in the course of the instruction, and to determine the correlation, if any, between these and the formation of math conceptualization.

Broader Impacts: This project should have significant direct impact on inclusive mathematics instruction at all grade levels for visually impaired students; provision of a sense of embodiment awareness to students who are blind has not hitherto been studied, and has the potential for empowering such students. Furthermore, understanding the channels for embodiment awareness will affect the design of future distance learning systems, and will afford insights on how best to provide embodiment cues to students in Internet-based instruction.

Deception is a pervasive feature of social life yet often goes undetected because deceivers capitalize on features of the interpersonal communication process. They adjust their verbal and nonverbal behavior over the course of an interaction in ways that evade detection. To uncover the complexities and dynamics of the communication processes that make successful deception or detection possible, collaborative research will be conducted by a multidisciplinary team of communication, linguistics, psychology, computer science, and management information systems researchers from the University of Arizona, University of Chicago, Michigan State University, and Rutgers University. They will be joined by international experts from the University of Cologne, Imperial College London, and University of Iceland. The project will develop a theoretical model of interpersonal deception that shifts emphasis from stable individual behaviors to dynamic interaction patterns. It will create five test beds by measuring the verbal and nonverbal features from video-recorded interpersonal communication experiments. The five experiments are a cheating experiment, a mock theft experiment, deceptive interviews, a group collaboration task, and a narration task. Measurement will consist of extensive automated and human-annotated measurement of visual, vocal, and verbal features. As part of the annotation work, the team will refine and validate software for automated measurement of nonverbal and verbal features. Theory-driven hypothesis tests and exploratory tests will be conducted on the measured communication behaviors to identify dynamic adaptation patterns using time-series, Bayesian-based Theme analyses, and artificial intelligence data mining techniques. The project will advance the scientific infrastructure for studying deception by forming the largest international and multidisciplinary research team of deception experts of its kind, integrating the knowledge and methods of multiple disciplines, refining computer-aided analysis of human communication, making progress in automating deception detection, and educating new investigators through laboratory exchanges. Society will benefit from a more valid picture of deception that can guide training and detection efforts in public, business, government, and security settings.

Members of different cultures may behave quite differently from one another when interacting face-to-face. Culture-specific aspects of speech and nonverbal behavior are signals that enable members of a culture to establish and maintain a sense of rapport with one another over intervals of interaction. Rapport, and the means by which conversation partners achieve it, is important to study systematically, because rapport is known to increase the likelihood of success of goal-directed interaction, and also to promote knowledge sharing and learning. Subtle cues signal engagement, endorsement, or appreciation. In the verbal channel, these include mirroring of word choices and of grammatical structures as well as vocal feedback. Similarly, many dimensions of nonverbal behavior such as posture, gaze, nods, and gesticulation, signal -- both to the conversation participants and to observers of them -- the extent to which the participants feel a sense of affiliation. A multidisciplinary team of psychologists, anthropologists, linguists, and computer scientists, will examine and compare such indices of rapport in natural interactions across members of three diverse language/cultural groups: Gulf/Iraqi Arabic-, Mexican Spanish-, and American English-speaking cultures.

An integral part of this project is to further develop technologies that enable micro-analyses of synchronized gesture and speech, both within individuals and across conversation partners. These technologies include computer interfaces for visualization, annotation, and analysis of complex, multimodal behaviors. The project will develop techniques to automatically recognize states of rapport and the verbal and nonverbal signals that lead to its disruption. The results of these examinations of human-human interaction will then be used to program behavioral repertoires for 'Embodied Conversational Agents' (ECAs). These are computer-generated, two-dimensional figures, human in appearance and capable of a range of verbal and nonverbal behaviors characteristic of listeners in interactions. The team will study human research participants in interaction with the ECAs, manipulating, in the ECAs, aspects of behavior identified as related to establishment, maintenance, and disruption of rapport, so as to observe the effects on human participants. The cross-cultural dimension of this comparative study will make it possible to identify what aspects of behavior are crucial for scaffolding successful intercultural interaction and communication among people of different cultures.

This research project will investigate the effects social organization, learning technologies, and their interrelationships, have on the geometric reasoning of PreK-3 students. Social organization is operationalized here as the strategic grouping of students into triads to engage in collaborative problem solving. Learning technologies are operationalized here as physical and virtual manipulative types (tangrams, pentominoes, geoboards) together with supporting software designed to facilitate students in accessing their mathematical knowledge. Based on findings from preliminary work in this area, it is proposed that learning technologies allow for fluid exchanges between peers, and transitions between physical and virtual manipulatives, significantly enhancing geometric reasoning. The current project presents a robust research and design framework that combines iterative cycles of field studies in the classroom with development and experimentation in the lab. The goal is to design, develop and prototype a tabletop computer interface that allows for unprecedented exchanges among student peers and transitions between manipulative types. The tabletop interface will serve first as the apparatus that facilitates experimental manipulations, later being used, through iterative refinement, as the learning technology platform by which students gain access to mathematical knowledge. The advancement proposed by the current work is that it places all interaction on the horizontal tabletop. This is critical in that it removes confounding effects of different seating and screen orientations from pilot study findings. Manipulations of geometric artifacts reside along two dimensions of research questions: the effects of social interaction and of learning technology on geometric reasoning. Hence, the proposed studies will allow investigation of different transitions between fully functional and fully operational thinking, functional and qualitative operational thinking, and qualitative and quantitative parametric thinking with the same interaction morphologies.

The proposed research has the potential to make significant contributions to the following intellectual domains: 1) child development -understanding children's development of mathematical reasoning in terms of learning trajectories 2) mathematics education - understanding teachers' use of advanced learning technologies and role in classroom instruction, and children's learning of mathematics using advanced learning technologies; 3) learning sciences - refining techniques for the analysis, design, and development of advanced learning technologies in use; 4) human-computer interaction -design of interfaces that aid learning to include gestural interaction, tangible manipulation, and horizontal displays; and 5) psycholinguistics - grounding of concepts developed in other domains in the classroom environment. Results from the research will also have much to say about the participation of girls in the mathematics classroom, a goal being to encourage rich discourse of all students. In addition, developed technologies may have implications for the education of special needs children as instructional interfaces can be modified and reconfigured to meet the diverse needs of this underrepresented group. A significant outcome of collaboration among these disciplines will be cutting-edge research opportunities for doctoral students and groundwork for the establishment of new curricula and, possibly, degree programs.