Jeffrey F. Cohn, Ph.D. - US grants

Although it is widely assumed that bidirectional influence accounts for the structure of early interactions, this hypothesis has never been adequately tested against alternate hypotheses. In order to evaluate this and alternate hypotheses, interactions between 54 mother-infant pairs--18 each at 3, 6, and 9 months--have been coded with a set of behavioral descriptors and a 1/4 sec time base. The interaction data will be analyzed with two separate sets of procedures that correspond to the principal ways in which behavior has been operationalized in studies of social interaction. In the first set, transition frequencies among mother-infant joint states will be analyzed with Thomas & Malone's (1979) Model 1. In the second, mother and infant scaled scores will each be scaled along an attentional/affective dimension, and these will be analyzed with Box-Jenkins bivariate time series techniques. Both sets of analyses will provide estimates of: infant sensitivity to own and other's prior response; and comparable estimates for the mother. The hypothesis of bidirectional influence will be supported at those ages for which infant and mother sensitivity to other's prior response is significant. Developmental changes in bidirectional influence will be analyzed with nonparametric techniques, for the Thomas & Malone parameter estimates, and ANOVA for the time series estimates. The proposed study will provide a comprehensive evaluation of the bidirectional and alternate hypotheses.

DESCRIPTION (Applicant's Abstract): Facial expression communicates information about emotional response and plays a critical role in the regulation of interpersonal behavior. Current human-observer based methods for measuring facial expression are labor intensive, qualitative, and difficult to standardize across laboratories and over time. To make feasible more rigorous, quantitative measurement of facial expression in diverse applications, we formed an interdisciplinary research group which covers expertise in facial expression analysis and image processing. In the funding period, we developed and demonstrated the first version of an automated system for measuring facial expression in digitized images. The system can discriminate nine combinations of FACS action units in the upper and lower face, quantity the timing and topography of action unit intensity in the brow region; and geometrically normalize image sequences within a range of plus or minus 20 degrees of out of-plane. In the competing renewal, we will increase the number of action unit combinations that are recognized, implement convergent methods of quantifying action unit intensity, increase the generalizability of action unit estimation to a wider range of image orientations, test facial image processing (FIP) in image sequences from directed facial action tasks and laboratory studies of emotion regulation, and facilitate the integration of FIP into existing data management and statistical analysis software for use by behavioral science researchers and clinicians. With these goals completed, FIP will eliminate the need for human observers in coding facial expression, promote standardize measurement, make possible the collection and processing of larger, more representative data sets, and open new areas of investigation and clinical application.

When humans converse, semantic verbal content is accompanied by vocal prosody (the emphasis and timing of speech), head nods, eye movements, eyebrow raises, and mouth expressions such as smiles. Coordination between conversants' movements and/or facial expressions can be observed when an action generated by one individual is predictive of a symmetric movement by another: symmetry formation. The interplay between such symmetry formation and subsequent symmetry breaking in nonverbal behavior is integral to the process of communication and is diagnostic of the dynamics of human social interaction. The PIs propose a model in which low level contributions from audition, vision, and proprioception (the perception of the angle of our joints) are combined in a mirror system that assists affective and semantic communication through the formation and breaking of symmetry between conversants' movements, facial expressions and vocal prosody. In the current project, naive participants will engage in dyadic (one-on-one) conversations with trained laboratory assistants over a closed-circuit video system that displays a computer reconstructed version of the lab assistant's head and face. Both the naive participant's and the lab assistant's motions, facial expressions, and vocalizations will be recorded. The visual and auditory stimuli available to the naive participant will be manipulated to provide specific hypothesis tests about the strength and timing of the effects of head movement, facial expression, and vocal prosody. The visual manipulation will be provided by a photorealistic reconstructed avatar head (i.e., a computer animation) driven partially by tracking the lab assistant's head and face, and partially from manipulation of timing and amplitude of the avatar's movement and facial expression. A combined differential equations and computational model for the dynamics of head movements and facial expression will be constructed and tested in real-time substitution for lab assistant's head motion or facial expression as realized by the avatar. The broader impact of this project falls into three main areas: enabling technology for the study of human and social dynamics, applications to the treatment of psychopathology, applications to human-computer interface design and educational technology. (1) These experiments will result in the advancement of methods for testing a wide variety of hypotheses in social interaction where the research question involves a manipulation of perceived social roles. (2) Automated analysis of facial expression provides on-line analysis of social interactions in small group, high stress settings in which emotion regulation is critical such as in residential psychiatric treatment centers and in psychotherapist-client interactions. The reliability, validity, and utility of psychiatric diagnosis, assessment of symptom severity, and response to treatment could be improved by efficient measurement of facial expression and related non-verbal behavior, such as head gesture and gaze. (3) Successful outcomes from the computational models may lead to the development of automated computer interfaces and tutoring systems that could respond to students' facial displays of confusion or understanding, and thereby guide more efficient instruction and learning.

Young infants typically form lasting, emotional attachments to their caregivers. The strength and type of these attachments are related to emotional well-being and cognitive development. This project will explore how face-to-face interactions between infants and adults contribute to this important aspect of child development.

During early interactions, infants and parents form expectations about one another. Will a smile be answered with a bigger smile, for example, or with no smile at all? If the parent is asked to stop interacting and just look at her infant, will the infant smile or vocalize in an attempt to repair the interaction? Do these early patterns of interaction predict the infant's later security of attachment--their ability to be comforted after a brief separation from the parent? To answer these questions, seventy-five infants and their mothers will participate in a standard "Face-To-Face/Still-Face"
procedure at four months. Their security of attachment will then be assessed at
12 months.

It is difficult to measure early interactive behavior-and human behavior more generally-objectively and efficiently. To address this challenge, the project's interdisciplinary team of psychological and computer scientists will implement automated, quantitative measurements of behavior in the Face-To-Face/Still-Face procedure. Automated facial image analysis and pattern recognition approaches will be used to produce objective, continuous measurements of infant and mother facial expression, head motion, gaze direction, and vocalizations. Precise measurement of this multimodal suite of infant and mother behaviors will be used to tackle a fundamental scientific problem: Modeling the structure of early interaction and its relation to later development.

This is a promising approach to understanding threats to typical development and learning associated with risk factors such as maternal depression and disorders such as autism. To maximize the project's impact, the team will make a database of audiovisual recordings, automated measurements, and pattern recognition and modeling software available to other scientists.

DESCRIPTION (provided by applicant): Facial expression has been a focus of emotion research for over a hundred years. In recent decades observations of facial expressions have yielded critical and dramatic insights about the etiology of psychopathology, and have proven capable of predicting treatment outcomes (see Ekman & Rosenberg, 2005). Despite these original striking findings, there has been surprisingly little follow-up work. The primary reason fr the lack of sustained research is that the most reliable manual systems for measuring facial expression often require considerable training and are labor intensive. Automated measurement using computer vision and machine learning seeks to address the need for valid, efficient, and reproducible measurement. Recent systems have shown promise in fairly small studies using posed behavior or structured contexts with confederates, or trained interviewers, or pre-trained (person-specific) face models. For automated coding to be applied in real-world settings, a large data base with ample variability in pose, head motion, skin color, gender, partial occlusion, and expression intensity is needed. We have developed a unique database that meets this need and the algorithms necessary to enable robust automated coding. The database consists of 720 participants in three-person groups engaged in a group formation task. In a preliminary study, we demonstrated that our algorithms can successfully code two key facial signals associated with human emotion in this relatively unconstrained context (Cohn & Sayette, 2010). To achieve efficient, accurate, and valid measurement of facial expression usable in research and clinical settings, we aim to 1) train and validate classifiers to achieve reliable facial expression detectin across this unprecedentedly large, diverse data set; 2) extend the previous person-specific methods to person-independent (generic) facial feature detection, tracking, and alignment; and 3) make these tools available for research and clinical use.

This is funding to support participation by about 8 graduate students from U.S. institutions, along with about 5 senior members of the ICMI community who will serve as mentors, in a Doctoral Consortium (workshop) to be held in conjunction with and immediately preceding the 16th International Conference on Multimodal Interaction (ICMI 2014), which will take place November 12-16, 2014, at Bogazici University in Istanbul, Turkey, and which is organized by the Association for Computing Machinery (ACM). The ICMI conference series is the premier international forum for multidisciplinary research on multimodal human-human and human-computer interaction, interfaces, and system development. The conference focuses on theoretical and empirical foundations, component technologies, and combined multimodal processing techniques that define the field of multimodal interaction analysis, interface design, and system development. Topics of special interest to the conference this year include: multimodal interaction processing; interactive systems and applications; modeling human communication patterns; data, evaluation and standards for multimodal interactive systems; and urban interactions. ICMI 2014 will feature a single-track main conference which includes: keynote speakers, technical full and short papers (including oral and poster presentations), special sessions, demonstrations, exhibits and doctoral spotlight papers. The ICMI 2014 proceedings will be published by ACM Press and included in the ACM Digital Library. As a further incentive for high-quality student participation ICMI 2014 will be awarding outstanding paper awards, with a special category for student papers. More information about the conference may be found online at http://icmi.acm.org/2014/.

The goal of the ICMI Doctoral Consortium is to provide PhD students with an opportunity to present their work to a group of mentors and peers from a diverse set of academic and industrial institutions, to receive feedback on their doctoral research plan and progress, and to build a cohort of young researchers interested in designing multimodal interfaces. Student participants will present their ongoing thesis research as a short talk at the Consortium and also as a poster at the conference Doctoral Spotlight Session. Following the fruitful experience of last year's ICMI conference and doctoral consortium, and with the goal of providing more opportunities for interaction between the students and senior members of the field, the program will once again include a lunch on the day of the workshop for students and mentors, a career panel that will provide the students and mentors the opportunity to ask and answer questions and discuss challenges and opportunities in the field, and a dinner that will provide the students with the opportunity to hold informal conversations among themselves as well as with the organizers and mentors.

The Doctoral Consortium will give student participants exposure to their new research community, both by presenting their own work and by observing and interacting with established professionals in the field. It will encourage students at this critical time in their careers to begin building a social support network of peers and mentors. The organizers will take steps proactively to achieve a diversity of research topics, disciplinary backgrounds, methodological approaches, and home institutions among the students. To further increase diversity, the organizers have committed that no more than two students will be invited from any given U.S. institution of higher learning, and when two are accepted one of them must be a woman.

This project will extend and sustain a widely-used data infrastructure for studying human emotion, hosted at the lead investigator's university and available to the research community. The first two versions of the dataset (BP4D and BP4D+) contain videos of people reacting to varied emotion-eliciting situations, their self-reported emotion, and expert annotations of their facial expression. Version 1, BP4D (n=41), has been used by over 100 research groups and supported a successful community competition around recognizing emotion. The second version (BP4D+) adds participants (n = 140), thermal imaging, and measures of peripheral physiology. The current project greatly broadens and extends this corpus to produce a new dataset (BP4D++) that enables deep-learning approaches, increases generalizability, and builds research infrastructure and community in computer and behavioral science. The collaborators will (1) increase participant diversity; 2) add videos of pairs of people interacting to the current mix of individual and interviewer-mediated video; 3) increase the number of participants to meet the demands of recent advances in "big data" approaches to machine learning; and 4) expand the size and scope of annotations in the videos. They will also involve the community through an oversight and coordinating consortium that includes researchers in computer vision, biometrics, robotics, and cognitive and behavioral science. The consortium will be composed of special interest groups that focus on various aspects of the corpus, including groups responsible for completing the needed annotations, generating meta-data, and expanding the database application scope. Having an infrastructure to support emotion recognition research matters because computer systems that interact with people (such as phone assistants or characters in virtual reality environments) will be more useful if they react appropriately to what people are doing, thinking, and feeling.

The team will triple the number of participants in the combined corpora to 540. They will develop a dyadic interaction task and capture data from 100 interacting dyads to support dynamic modeling of interpersonal influence across expressive behavior and physiology, as well as analysis of emotional synchrony. They will increase the density of facial annotations to about 15 million frames in total, allowing the database to become sufficiently large to support deep-learning approaches to multimodal emotion detection. These annotations will be accomplished through a hybrid approach that combines expert coding using the Facial Action Coding System, automated face analysis, and crowdsourcing with expert input from the research community. Finally, the recorded data will be augmented with a wide range of meta-data derived from 2D videos, 3D videos, thermal videos, and physiological signals. To ensure the community is involved in sustaining the infrastructure, in addition to the governance consortium described above, the investigators will involve the community in jointly building both APIs that allow adding meta-data and annotations and tools to support the submission and evaluation of new recognition algorithms, then organizing community-wide competitions using those tools. The research team will also reach out to new research communities around health computing, biometrics, and affective computing to widen the utility of the enhanced infrastructure, grow the community of expert annotators through training workshops, and build an educational community around the infrastructure that facilitates the development and sharing of course materials that use it. Long-term, the infrastructure will be funded through a combination of commercial licensing and support from the lead university's system administration group.

Using multimodal indicators, this project will develop a novel computational framework that models individual and interpersonal behavior in relation to process and outcomes in psychotherapy and other interpersonal contexts. The unique aspect of the project is the explicit joint and dyadic modeling of individuals' multimodal behaviors to holistically understand the system of the dyad. This research will pave the way to a better understanding of the dyadic behavior dynamics in psychotherapy and beyond. The project will build the computational foundations to predict process and outcomes, and more broadly inform behavioral science: The project will (1) contribute to knowledge about the psychotherapeutic process by identifying and characterizing behavior indicators with respect to process and outcome measures; (2) deepen our understanding of dyadic coordination dynamics that contribute to strong working alliance between clients and therapists; (3) make available to the research and clinical communities the Dyadic Behavior Informatics framework and Behavior Indicator Knowledgebase for use in other settings; and (4) establish the foundation for novel education and training materials and interventions. The knowledge and computational tools developed as part of this project will impact computing and behavioral science and applied domains more broadly.

This project will advance understanding of dyadic behavioral dynamics by developing computational representations that can model fine-grained dyadic coordination between individuals and new algorithms that can model multi-level dynamics. Central to this research effort is the creation of the Behavior Indicator Knowledgebase (BIK) that will summarize discovered knowledge about significant and validated dyadic behavior indicators. While this work could have profound impact on behavioral and social science as a whole, the project specifically focuses on understanding the dynamics that predict process and outcome variables in psychotherapy. The project identifies five fundamental research challenges and presents a plan to address them directly: (1) Acquire a large, dyadic, and multimodal dataset of 64 patients with distress disorders seen over 8 psychotherapy sessions; (2) Create multimodal behavior indicators which can model the within session dynamics of the client or therapist; (3) Develop new dyadic behavior indicators that explicitly model the client-therapist coordination, (4) Develop abstract dyadic behavior representations that can learn the fine-grained dynamics between client and therapist behaviors; and, (5) Validate the computational representations (embeddings) and prediction models by assessing their impact on the predictive power of process and outcome measures in psychotherapy and assess generalizability beyond psychotherapy.