David McNeill - US grants

The symptomatology of aphasia after left-hemisphere damage should not distract us from the effects of right-hemisphere damage on the verbal performances of individuals who suffer it. These effects are subtle and are not manifested in a single subsystem like language, but affect the patient's ability to represent complex structures such as narratives and other forms of discourse. By considering complex performances (narratives, living-space descriptions, and descriptions of networks), plus tracking the nonverbal components of the performance (the speaker's gestures that accompany speech), we can study the deficits due to right-hemisphere damage in a clear and rigorous way. Our hypothesis is that normal speaking -- with a dual presentation of verbal and imagistic forms of representations - - involves the cooperative interaction of both hemispheres, each making its own specific contribution. Our hypothesis recognizes that speech and gesture constitute, at some level, a single system. Spatial representations thus can condition linguistic choices and vice versa. Patients with right hemisphere damage syndrome have seemingly unimpaired speech, but manifest poor story structure and gesture. There can be dissociation such that the strictly linguistic contributions to discourse structure occur in the absence of the appropriate imagery. This produces an 'empty' linguistic performance, not backed by the visuospatial representations that are part of normal metanarrative structure. In contrast, in left hemisphere patients, there are intact visuospatial representations, however imperfectly they may be expressed in linguistic terms, and these show themselves in both gesture and speech. Thus, according to our hypothesis, the narrative performance of left hemisphere patients may be underestimated by a too-narrow focus on the linguistic details of narrative speech and the performance of right hemisphere patients, due to the same narrow focus, may be overestimated . The relevance of this research for health lies in the understanding it will offer the contribution of the right and left hemispheres to language, gesture and discourse; of the role it tests of spatial imagistic representations in the formation of narrative and other discourse beyond specific linguistic choices; and of possible new approaches to therapy and evaluation of hemisphere damaged patients.

When we speak, our embodied behavior of gesture, gaze, posture, and facial expression are brought into the service of the communicative process. This research is grounded in psycholinguistic theories of multimodal human communication. One path from multimodal behavior to language is bridged by the underlying mental imagery. This spatial imagery, for a speaker, relates not to the elements of syntax, but to the units of thought that drive the expression (vocal utterance and visible display). Gestures reveal the focal points of the accompanying utterances, and relate to the meaning of the newsworthy elements of the unfolding discourse. This is the underlying premise of the current project, in which the PI will focus on math discourse and education for blind students.

Mathematical reasoning is rich in spatial imagery that is revealed in gesture. Furthermore, gesture has the capacity to create images of the math concepts that serve as "objects of contemplation." When a graphic/illustration is available for math instruction, the discourse stream is typically situated with gestures of spatial reference into the graphic. Research with individuals who are blind suggests that they have remarkable capacity for visual imagery, memory, and conceptualization. However, students who are blind tend to lag behind sighted students in mathematics education. The PI posits that a significant impediment to math instruction for students who are blind lies in the lack of visual access to the embodiment of the instructor. He notes that we have, in such students, a population that is able to access graphical content (through tactile image technology), but is not visually aware of the embodied behavior of the interlocutor. He proposes to remedy this problem by affording blind students a sense of embodiment awareness, with the help of augmentative communication approaches that employ tactile devices to provide elements of embodiment awareness. To this end, the PI will perform a series of "perception and action" experiments to assess the efficacy of such devices, followed by a second series of experiments with blind and sighted students in mathematics instruction, to be captured on video, and with pre- and post- tests, to assess the quality and quantity of imagistic content both conveyed by the teacher and by the student in the course of the instruction, and to determine the correlation, if any, between these and the formation of math conceptualization.

Broader Impacts: This project should have significant direct impact on inclusive mathematics instruction at all grade levels for visually impaired students; provision of a sense of embodiment awareness to students who are blind has not hitherto been studied, and has the potential for empowering such students. Furthermore, understanding the channels for embodiment awareness will affect the design of future distance learning systems, and will afford insights on how best to provide embodiment cues to students in Internet-based instruction.

This research project will investigate the effects social organization, learning technologies, and their interrelationships, have on the geometric reasoning of PreK-3 students. Social organization is operationalized here as the strategic grouping of students into triads to engage in collaborative problem solving. Learning technologies are operationalized here as physical and virtual manipulative types (tangrams, pentominoes, geoboards) together with supporting software designed to facilitate students in accessing their mathematical knowledge. Based on findings from preliminary work in this area, it is proposed that learning technologies allow for fluid exchanges between peers, and transitions between physical and virtual manipulatives, significantly enhancing geometric reasoning. The current project presents a robust research and design framework that combines iterative cycles of field studies in the classroom with development and experimentation in the lab. The goal is to design, develop and prototype a tabletop computer interface that allows for unprecedented exchanges among student peers and transitions between manipulative types. The tabletop interface will serve first as the apparatus that facilitates experimental manipulations, later being used, through iterative refinement, as the learning technology platform by which students gain access to mathematical knowledge. The advancement proposed by the current work is that it places all interaction on the horizontal tabletop. This is critical in that it removes confounding effects of different seating and screen orientations from pilot study findings. Manipulations of geometric artifacts reside along two dimensions of research questions: the effects of social interaction and of learning technology on geometric reasoning. Hence, the proposed studies will allow investigation of different transitions between fully functional and fully operational thinking, functional and qualitative operational thinking, and qualitative and quantitative parametric thinking with the same interaction morphologies.

The proposed research has the potential to make significant contributions to the following intellectual domains: 1) child development -understanding children's development of mathematical reasoning in terms of learning trajectories 2) mathematics education - understanding teachers' use of advanced learning technologies and role in classroom instruction, and children's learning of mathematics using advanced learning technologies; 3) learning sciences - refining techniques for the analysis, design, and development of advanced learning technologies in use; 4) human-computer interaction -design of interfaces that aid learning to include gestural interaction, tangible manipulation, and horizontal displays; and 5) psycholinguistics - grounding of concepts developed in other domains in the classroom environment. Results from the research will also have much to say about the participation of girls in the mathematics classroom, a goal being to encourage rich discourse of all students. In addition, developed technologies may have implications for the education of special needs children as instructional interfaces can be modified and reconfigured to meet the diverse needs of this underrepresented group. A significant outcome of collaboration among these disciplines will be cutting-edge research opportunities for doctoral students and groundwork for the establishment of new curricula and, possibly, degree programs.