Daniel S. Messinger - US grants

Before the onset of speech, an index of the developing communicative significance of infant social actions is their temporal coordination. One type of temporal coordination occurs when an action (e.g., a vocalization) both begins and ends during the course of another (e.g., a gaze at mother). This pattern of temporal embedding suggests the two actions are part of a single expressive event. The proposed project will use a new software simulation bootstrapping procedure to document embedding and other patterns in the temporal coordination of infant social actions. 50 infants will be observed longitudinally at 3, 6, and 9 months interacting with their mothers in a face-to-face still- face procedure. Descriptive coding will be used to categorize infant vocalizations and facial expressions by affective tone. Direction of infant gaze will be coded as at mother's face/eyes or away. Patterns of reciprocal embedding are hypothesized for actions in all pairs of behavioral modalities. Infants are expected, for example, to begin and end facial expressions in the course of a gaze at mother's face; but they are also expected to begin and end gazes at mother in the course of a facial expression. Software incorporating a bootstrapping procedure simulates the temporal progression of infant actions independently within each modality, creating multiple simulated sessions with random levels of between modality coordination. It will indicate whether infants reciprocally embed social actions at greater than chance levels. Increases in temporal embedding are expected with age as infants become more competent communicators. Decreases in embedding are expected when infant's are frustrated by maternal non-responsiveness in the still face procedure. The relative strength of embedded patterns involving effectively positive and negative actions will be contrasted to shed light on their respective expressive clarity. In sum, a new approach will be used to determine whether embedded events occur at greater than chance levels 1) overall, 2) during interaction rather than a maternal still-face, 3) at different ages, and 4) among different effectively toned pairs of social actions. A long term objective of the project is to gather pilot data for future investigations of temporal coordination between infant and mother (e.g., turn-taking). This will open the way for exploration of coordination deficits stemming from infant communicative disorders such as autism and Down syndrome, as well as coordination deficits stemming from maternal risk factors such as depression.

DESCRIPTION (provided by applicant): Intense emotional expressions - both positive and negative - characterize preverbal communication, influencing interaction with caregivers and later development. Multiple methods will be used to assess whether emotional intensity is indexed by facial markers that occur during both positive (smile) and negative (cry-face) facial expressions. These hypothesized intensity markers are stronger lip corner actions, the raising of the cheeks around the eyes, and mouth opening. One hundred and six four-month-olds have engaged in a standardized videotaped protocol - the face-to-face/still-face protocol (FFSFP) - that reliably elicits a wide range of positive and negative facial expressions. The duration of smiles and cry-faces that do and do not involve each of the three hypothesized intensity markers will be reliably coded from close-up videotaped images of the infants' faces using an anatomically-based coding stem (FACS/BabyFACS). The facial intensity markers are expected to occur simultaneously during smiles and during cry-faces, an index of internal validity. Infants prenatally exposed to cocaine who are at risk for non-optimal emotion regulation (n=47) are hypothesized to show smiles with a smaller number of intensity markers and cry-faces with a greater number of markers. Smiles and cry-faces involving a greater number of hypothesized intensity markers are expected, respectively, during more positive and negative episodes of the FFSFP. Smiles and cry-faces with higher numbers of intensity markers are hypothesized to be more likely to co-occur with affectively concordant vocalizations and with more rapid heart rate than expressions with fewer markers. Naive observers are hypothesized to rate smiles and cry-faces involving a higher number of intensity markers as, respectively, more emotionally positive and more negative than expressions with fewer markers. In sum, empirically validated markers of the intensity of early positive and negative emotional expression will be documented by investigating whether expressions involving a greater number of overlapping markers occur during affectively appropriate situations in conjunction with other indices of affective intensity among more and less at-risk infants, and by determining whether these expressions are perceived by native observers as being especially emotionally intense.

Infant smiles can predict later adaptive functioning, but little is known about the temporal course of infant smiles or their perceived emotional intensity. This collaborative project combines computer-based measurements of infant smiles with parents' ratings of those smiles. The project goals are to understand how infants smile, and to document the features that make infant smiles appear more or less joyful.

To understand how infants smile, the smile strength and co-occurring mouth opening of 50 six-month-old infants' smiles will be measured using computer software (CMU/Pitt Automated Facial Image Analysis, v. IV). These objective measurements will be used to document how infant smile form, peak, and dissolve in time. That is, we will create real-time portraits of smiles to more fully understand positive emotional functioning. Statistical associations between smile strength and mouth opening will be examined, as will differences - both between smiles and between infants - in how strongly these components of smiles are associated. Complementary graphical and statistical approaches will encourage dynamic accounts of emotional functioning and new avenues of research on the role of emotion in learning and development.

The project uses rating studies to document features that make infant smiles appear more or less joyful. Both parents of infants and undergraduates will be asked to rate the emotional intensity of both still images and video clips of smiles. Automated measurements of smile strength and mouth opening will then be used to predict ratings of positive emotional intensity. Positive findings would validate automated measurements of specific features of infant smiles as indices of infant joyful emotional intensity.

In this project, leading infant emotion researchers will collaborate with a cutting-edge automated facial measurement team to investigate infants' smiles. Infants use positive facial expressions to express emotions and communicate with others. Basic knowledge of the temporal course of in vivo infant emotional expressions promises an objective view of normative and compromised emotional functioning and development. To that end, the acquired database of digitized facial expressions and their automated measurements will be made available to requesting investigators. These objective descriptions will be complemented by parental and naive observers' perceptions of the emotional intensity of these expressions. This will validate the emotional significance of the automated measurements, and contribute to the development of a tool for automated coding that has the potential for broad impact.

DESCRIPTION (provided by applicant): This project will use developmental patterns of early emotional functioning and communication to predict social, behavioral, and cognitive functioning in the second and third year of life. Contrasts between 100 typically developing infant siblings and 100 infant siblings of children with Autism Spectrum Disorder (ASD) - who are at risk for a range of disruptions of emotion, communication, and cognition - will illuminate these developmental processes from a developmental psychopathology perspective. In the first year of life, associations between temperamental (Lab-TAB) and communicative (face-to-face/still-face) assessments of positive and negative emotion, cerebral indices of approach motivation (EEG laterality), and the development of communicative intentionality (infant joint attention) will be determined. These constructs will be used to predict a set of developmental^ linked processes including dyadic relationship quality, parent rated behavioral difficulties and strengths, standardized measures of linguistic and cognitive functioning, and ASD- related symptomatology in the second and third year of life. Relationship quality is a central construct, encompassing security of attachment, parental warmth and control, and child cooperation and warmeth in a range of social situations. Relationship quality is expected to mediate pathways to all outcomes with the exception of ASD symptomatology. Children with high levels of ASD symptomatology are expected to show deficits in both early measures emotional communication and later measures of social, communicative, and cognitive functioning. ASD symptomatology is expected to be directly related to early emotional functioning. This proposal utilizes multi-method assessment of emotional functioning early in infancy to confront the task of predicting intentional communication and later indices of healthy interaction, and competent linguistic and cognitive functioning. The models tested will shed specific light on the prediction and prevention of health- related outcomes including behavior problems, poor relationship quality, and ASD symptomatology in an at- risk and typically developing sample. Examination of predictive associations in typically developing infants and infant siblings of children with ASD will shed light on pathways to healthy development, psychopathologic deviations from those pathways, and potential intervention strategies for infants at risk for developmental difficulties.

[unreadable] DESCRIPTION (provided by applicant): A tool for obtaining efficient, transparent, replicable measurement of social behavior would increase understanding of pathways to healthy child development. This application will validate real-time, continuous measurements of social behavior. Measurements will be made by multiple naive raters (using a joystick), whose ratings are aggregated to increase their precision and generalizability. Continuous ratings will be conducted using four already-acquired expertly coded data-sets. Ratings will include the emotional valence of high-risk infants, the joint-engagement of middle-class dyads, the symbolic play of hearing impaired children, and the negativity of families with children who have behavior problems. The application will examine the reliability of measurements made using this continuous rating software, and the convergent, concurrent, and predictive validity of these ratings. High concordance between naive raters' continuous ratings and the reliable measurements already obtained by expert coders would indicate convergent validity. Continuous ratings of behavioral qualities are also expected to differ between groups (e.g., normal vs. hearing impaired), and to change with age (i.e. 9, 18, & 24 months) and between situations (i.e., face-to-face vs. still-face) in a manner parallel to those seen in the expert coded measures. Significant associations between continuous ratings (such as time-series of infant-mother perceived emotion) made during the same assessment will indicate concurrent validity. Significant associations between continuous ratings and later outcome (such as between joint engagement and language outcome) would indicate predictive validity. Encouraging healthy development involves understanding the social behavior of typical and at-risk children interacting with their families. Current behavioral measurement systems are typically complex and time-consuming, which creates barriers to the replication of results. To maximize public health impact, a user-adjustable version of the continuous rating software will be made available for download by investigators. This will allow investigations of observers' continuous ratings of a plethora of health-relevant constructs. With this software, naive raters are provided with brief, precise descriptions of constructs, which enhances the relevance of findings to health providers and policy makers because results will reflect a scientifically defined, simple view of the construct measured. [unreadable] [unreadable] [unreadable]

Last Modified Date: 08/04/08 Last Modified By: C.S. George Lee

Abstract
The goal of this project is to make progress on computational problems that elude the most sophisticated computers and Artificial Intelligence approaches but that infants solve seamlessly during their first year of life. To this end we will develop a robot whose sensors and actuators approximate the levels of complexity of human infants. The goal is for this robot to learn and develop autonomously a key set of sensory-motor and communicative skills typical of 1-year-old infants. The project will be grounded in developmental research with human infants, using motion capture and computer vision technology to characterize the statistics of early physical and social interaction. An important goal of this project is to foster the conceptual shifts needed to rigorously think, explore, and formalize intelligent architectures that learn and develop autonomously by interaction with the physical and social worlds. The project may also open new avenues to the computational study of infant development and potentially offer new clues for the understanding of developmental disorders such as autism and Williams syndrome.

DESCRIPTION (provided by applicant): Infant siblings of children with ASD exhibit substantial variability in their social-communicative outcomes: some manifest behaviors consistent with an autism spectrum diagnosis, others exhibit less severe symptoms such as language delay, and others evidence no detectable disorder. This cross-site, Vanderbilt- Miami project will examine the extent to which early impairments in specific attentional and affective mechanisms - both of which are putative core deficits of autism - explain the behavioral heterogeneity that is observed in these high-risk children. We focus specifically on the domains of attention coordination and positive affective competencies, and examine their development across the first two years of postnatal life (i.e., from 6 to 18 months) and their relation to later social and communicative outcomes at age 24 and 36 months. The overarching hypotheses are that there will be developmental continuity in these domains across the first two years, that early differences in these domains will be evident between groups at high- and low-risk for ASD, and that early impairments will predict individual differences in social and communicative outcomes observed in later-born ASD siblings. Specific areas of investigation include the relation between positive affective competencies in dyadic and triadic contexts across the first year of life, the contributions of early positive affective competencies and attention coordination to the emergence of positive joint attention competencies in the second year of life, and the extent to which individual differences in early positive affective competencies and attention coordination contribute to later autism symptomatology. This prospective longitudinal project will compare developmental trajectories and outcomes in 200 infant siblings of children diagnosed with ASD (Sib-ASD) and 100 infant siblings of typically developing children (Sib-TD). A split panel longitudinal design will allow us to examine development both within- and across- age-group cohorts from 6 months to 36 months of age. This project employs novel methods and measurement of infant behavior including electrophysiological and eye tracking measures, measures of positive affect communication and representation, and longitudinal measures of sharing positive emotion about objects with others. Longitudinal structural equation modeling and mixed modeling procedures are utilized to assess differences in the development of typically developing and at- risk siblings and to predict heterogeneity in the outcomes of the at-risk siblings of children with ASD. The project employs a developmental psychopathology perspective to shed light on normative and disturbed longitudinal pathways toward heterogenous outcomes. PUBLIC HEALTH RELEVANCE: The prospective study of younger siblings of children with ASD offers an ideal opportunity to learn about the earliest signs of autism as well as the development, manifestation, and boundaries of the broader autism phenotype. This information is critical for understanding the causes of autism and for developing more effective and targeted treatments.

Young infants typically form lasting, emotional attachments to their caregivers. The strength and type of these attachments are related to emotional well-being and cognitive development. This project will explore how face-to-face interactions between infants and adults contribute to this important aspect of child development.

During early interactions, infants and parents form expectations about one another. Will a smile be answered with a bigger smile, for example, or with no smile at all? If the parent is asked to stop interacting and just look at her infant, will the infant smile or vocalize in an attempt to repair the interaction? Do these early patterns of interaction predict the infant's later security of attachment--their ability to be comforted after a brief separation from the parent? To answer these questions, seventy-five infants and their mothers will participate in a standard "Face-To-Face/Still-Face"
procedure at four months. Their security of attachment will then be assessed at
12 months.

It is difficult to measure early interactive behavior-and human behavior more generally-objectively and efficiently. To address this challenge, the project's interdisciplinary team of psychological and computer scientists will implement automated, quantitative measurements of behavior in the Face-To-Face/Still-Face procedure. Automated facial image analysis and pattern recognition approaches will be used to produce objective, continuous measurements of infant and mother facial expression, head motion, gaze direction, and vocalizations. Precise measurement of this multimodal suite of infant and mother behaviors will be used to tackle a fundamental scientific problem: Modeling the structure of early interaction and its relation to later development.

This is a promising approach to understanding threats to typical development and learning associated with risk factors such as maternal depression and disorders such as autism. To maximize the project's impact, the team will make a database of audiovisual recordings, automated measurements, and pattern recognition and modeling software available to other scientists.

When infants look at someone's face or at an object, they are learning about the world. But neither parents nor scientists know how long an infant will look at a given face or object. This study tests the hypothesis that the durations of an infant's previous looks at a target predict the duration of the next look at the target. This is called the temporal dependency hypothesis. It suggests that infants regulate their own behavior to learn from their environment early in life. The temporal dependency hypothesis will be evaluated in a multi-ethnic group of infants from the Southeastern United States. During several visits in the first year of life, individual infant looks will be measured as infants interact with their mothers, and watch videos of people and objects. During each of these activities, individual infants may have a preferred level of temporal dependency that indicates how strongly their past looks predict their next looks. Greater temporal dependency is believed to reflect greater self-regulation. The project will assess whether greater temporal dependency predicts higher levels of communication and self-regulation at a year and a half of age.

The project unites research on social and cognitive development to understand the behavioral bases of visual exploration. It tests a hypothesis that may characterize other behaviors relevant to children's learning such as reaction times. Results will inform understanding of developmental disorders such as autism, and the development of new technologies such as robots with realistic and effective looking behaviors. To maximize this impact, project resources such as digital recordings of infant behavior will be shared with other researchers.

DESCRIPTION (provided by applicant): Significance. Computational modeling is central to a rigorous understanding of the development of the child's first social relationships. The project will address this challenge by modeling longitudinal change in the dynamics of early social interactions. Modeling will integrate objective (automated) measurements of emotion and attention and common genetic variants relevant to those constructs. Innovation. Objective measurement of behavior will involve the automated modeling and classification of the physical properties of communicative signals-such as facial expressions and vocalizations. Dynamic models of self-regulation and interactive influence during dyadic interaction will utilize precise measurements of expressive behavior as moderated by genetic markers associated with dopaminergic and serotonergic functioning. The interdisciplinary team includes investigators including from developmental and quantitative psychology, genetics, affective computing, computer vision, and physics who model dynamic interactive processes at a variety of time scales. Approach. Infant-mother interaction, its perturbation, and its development, will be investigated using the Face-to-Face/Still-Face (FFSF) procedure at 2, 4, 6, and 8 months. Facial modeling, head, and arm/hand modeling will be used to conduct objective measurements of a multimodal suite of interactive behaviors including facial expression, gaze direction, head movement, tickling, and vocalization. Models will be trained and evaluated with respect to expert coding and non-experts' perceptions of emotional valence constructs. Dynamic approaches to time-series modeling will focus on the development of self-regulation and interactive influence. Inverse optimal control modeling will be used to infer infant and mother preferences for particular dyadic states given observed patterns of behavior. The context-dependence of these parameters will be assessed with respect to the perturbation introduced by the still-face (a brief period of investigator-requested adult non-responsivity). Individual differences in infant and mother behavioral parameters will be modeled with respect to genetic indices of infant and mother dopaminergic and serotonergic function. Modeling algorithms, measurement software, and coded recordings will be shared with the scientific community to catalyze progress in the understanding of behavioral systems. These efforts will increase understanding of pathways to healthy cognitive and socio-emotional development, and shed light on the potential for change that will inform early intervention efforts.

This interdisciplinary research project will investigate the development of children's social networks using real-time observations of classroom behavior. Continuous measurement of children's movements in the classroom will be complemented with automated analyses of audio recordings and human descriptions of social contacts. Children will be followed from toddler through pre-school to pre-kindergarten in order to conduct a longitudinal investigation of the origin and development of their social networks. Interactions will be observed across classroom activities and will include teachers, allowing for a comprehensive model of classroom ecology and gender segregation patterns. The project will enhance understanding about the micro-geography of educational practice to determine how student location is distributed with respect to the pedagogic structuring of classroom space, such as which children visit the book area together. The findings will provide a knowledge base designed to address deficits in school engagement that characterize the ethnically diverse and economically disadvantaged students who will be participating in the project. Project resources and de-identified data will be disseminated to the research community to facilitate additional discovery.

This project will unite network scientists from physics with developmental and school psychologists to investigate the development and dynamic functioning of social networks. Quantitative network models of social dynamics will be infused with information about the role that child characteristics like gender and ethnicity play in the formation of classroom social groups. Agent-based modeling will synthesize movement and social contact dynamics. Network models will be parameterized with fission-fusion equations to predict changes in children's developing social networks such as the size and gender composition of groups of interacting children. Multilevel models will capture within- and between- year longitudinal changes in sociality for individual children and for the network overall. These complementary models will specify the processes through which children's social groups become progressively larger and more interconnected over development. This project is supported through the NSF Interdisciplinary Behavioral and Social Sciences Research (IBSS) competition.

Project Summary This Administrative Supplement proposes implementation of a multimodal data pipeline to support machine learning of child language production in complex naturalistic environments. The Supplement builds on the parent R01 (DC018542) that gathers objective, longitudinal data to capture the vocal interactions of children with hearing loss (HL). Even with cochlear implantation, HL is a life-altering condition with high social costs. Inclusion of children with HL and typically hearing (TH) peers in preschool classrooms is a national standard, but it is not clear how early vocal interaction contributes to the language development of children with HL and their TH peers. The parent R01 employs computational models of child location and orientation to indicate when children are in social contact with their peers and teachers. An additional strategy for pursuing the broad goals of the R01? identifying interactive contexts in which children produce phonemically complex vocalizations and interactive speech?is machine learning. Machine learning algorithms can determine the contextual, individual, and interactive factors that predict children?s vocalizations and vocal interactions. However, the parent R01 does not propose machine learning, nor are data disseminated in a format designed to facilitate machine learning. To facilitate machine learning in the classroom, a rigorous diarization process is required to determine speaker identity, which is operationalized as the likelihood that each vocalization was spoken by a given child or teacher. We will integrate audio processing of each target child and teacher?s first-person audio recording with processing of their interactive partners? recordings. The influence of partner recordings will be determined by their physical distance and orientation relative to the target. This will yield a weighted speaker identification score for each vocalization. For 25% of the sample, the algorithmic score will be compared to speaker identification provided by trained coders to quantify intersystem reliability. Processed datasets will include 7,160 hours of multimodal recordings of child and teacher movement in classrooms synchronized with continuously recorded, child- and teacher-specific (first-person) audio recordings. De-identified output data will characterize vocalizations with respect to algorithmically computed speaker identification probabilities, coder-identified speaker identity (25% of sample), phonemic complexity and audio characteristics (e.g., fundamental frequency), as well as the position and relative orientation of all individuals in the classroom, and child demographics (including characterizations of HL). Over the course of the supplement, output data, Python processing code, and metadata descriptions of the processing pipeline will be disseminated in dedicated distribution portals including Github, Kaggle, and the UCI repository. Recordings will be released to certified investigators via NIH-funded repositories such as Databrary and Homebank.

Project Summary This project uses objectively measured, longitudinal data to capture the vocal interactions of children with hearing loss in order to understand everyday factors that facilitate their language development. Significance. Hearing loss (HL) is associated with delays and difficulties in social and language development. Even with cochlear implantation (CI), HL is a life-altering condition with high social costs. Inclusion of children with HL and typically hearing (TH) peers in preschool classrooms is a national standard, but little is known about how early interaction contributes to the language development of these children. The project premise, supported by preliminary data, holds that language-mediated interactions with peers and teachers in inclusive classrooms positively influences the language development of both children with HL and their TH peers. Innovation. Temporally intensive, objective measurements of child and teacher movement in classrooms will be synchronized with automated analysis of continuously recorded, child-specific audio recordings. Computational modeling of child location and orientation within classrooms will indicate when children are in social contact with their peers and teachers, fertile periods for language-mediated interaction. Approach. Participants will include 125 children (55 with hearing loss who have cochlear implants or hearing aids, and 70 with typical hearing) who are enrolled in three inclusive oral language classrooms. Children will be observed longitudinally in an early preschool (2-year-old), preschool (3-year-old), and pre-kindergarten (4-year-old) classes. Each class will be observed once per year over the 5 year project. (15 unique classes). Objective measurements of social contact and language use occur twice a month while standardized assessments of receptive/expressive language competencies occur at the beginning and end of the school year. Aim 1. Determine whether exposure to partner speech during periods of social contact longitudinally predicts speech to those partners during social contact. Overall, higher levels of socially mediated exposure to peer and teacher speech are hypothesized to predict higher speech to those partners. Socially mediated exposure to more vocally responsive partner speech and more phonemically complex speech is hypothesized to yield higher rates of more phonemically complex child speech. Aim 2. Determine the longitudinal influence of speech to partners during social contact on the development of children's language competencies. Children's speech to peers and teachers?particularly more phonemically complex speech?is hypothesized to be associated with increases in their assessed language competencies. Aim 3. Determine whether hearing loss directly or indirectly impacts the longitudinal development of speech to partners during social contact and assessed language competencies. In addition to testing for differences between children with HL and TH, Aim 3 explores whether associations between socially mediated vocal interactions and the development of language competencies differs among children with HL on the basis of factors such as hearing age (time since augmented hearing). Aim 3 also explores the role of other individual differences such as biological sex and SES indices on patterns of interactive speech with partners and assessed language competencies. Rigor and reproducibility will be enhanced by the use of objective measurement, data-driven computational models, and dissemination of study tools and de-identified data.