Chen Yu - US grants

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

Children begin to comprehend words at 9 months. They say their first word at around 12 months. The pace of vocabulary learning then accelerates so that by 24 to 30 months, children add words at the staggering rate of 5 to 9 new words per day. There have been many studies focusing on documenting developmental progress in early language acquisition, and most theories of learning derived from those studies have focused on macro level descriptions that sound like explanations, such as "the mother tried to elicit the child's attention by waving the toy." These descriptions may capture higher-level human behaviors, but they fall short of a mechanistic account of how word learning works in real time. Toddlers learn words through millisecond by millisecond, second by second, and minute by minute events that are generated by actively engaging in the world, with objects, and with their social partners. But very little is known about how any of this works in real time and in the cluttered context of the real world interactions of toddlers and parents, contexts typically characterized by many interesting objects, shifts in attention by each participant, and goals (beyond teaching and learning words). In light of this, the series of experiments in this project will provide a systematic study of child-parent interaction and learning as coupled complex systems. The child's actions (head and eye movements, hand movements, picking up objects) create within the child dynamic dependencies of looking, seeing, touching and feeling. Each moment of perceptual and motor activity by the learner determines the next -- a head turn determines what is seen next, which may determine what is reached for and brought close to the eyes, which selects and generates the next view. Thus, the learner is a dynamic complex system. But the toddler is not alone when learning new words. Instead, a mature partner -- who is also a complex multimodal system -- offers words, gestures and actions. Critically, the streams of touches, sights and sounds from two participants are closely coupled, with one agent shaping the experiences and behaviors of the other. The study will measure the dynamic multimodal behavioral patterns within and across social partners as children and parents actively engage with and talk about objects in everyday contexts. The project will collect multiple streams of high-resolution, high-quality video and speech data from both participants. The dense and rich streams of multimodal data are useful only to the degree that one can find meaningful patterns in those dynamic streams that bring new insights into real-time learning events. To this end, the project will develop new methods of data analysis, visualization and data mining to quantify fine-grained behavioral patterns within an individual's cognitive, perceptual and motor systems and across social partners. This constitutes a significant advance in theoretical approaches to early word learning and one that also has broad applications. Measuring interaction patterns within and between complex systems is a critical problem across science -- from cells, to brains, to coupled physical systems, to human-computer interaction, to groups of animals, to teams of people. Thus, this research will bring new methods and analytic tools for measuring the information in coupled interactive systems.

Understanding learning mechanisms in the context of a dynamic, everyday learning environment is essential to understanding typical development, individual differences, and atypical development. Designing effective procedures to benefit children with developmental delays requires a principled understanding of that dynamic environment as it relates to the cognitive learning system. Thus, the work will provide scientists, educators, and parents with an understanding of children's early cognitive processes and general principles to facilitate child-parent social interaction and early language learning. Moreover, building anthropomorphic machines that can acquire language automatically may be best accomplished by emulating how toddlers learn language. Artificial intelligence systems with human-like language skills have important utilities in real-world applications. Finally, this approach is methodologically novel. Not only will it provide new findings, but the research will be a proving ground for the development and invention of these new techniques -- techniques that may be applied in many different domains of social and behavioral studies, such as typical and atypical cognitive development, collaboration and joint problem solving, and adult social interactions.

Human visual object recognition is fast and robust. People can recognize a large number of visual objects in complex scenes, from varied views, and in less than optimal circumstances. This ability underlies many advanced human skills, including tool use, reading, and navigation. Artificial intelligence devices do not yet approach the level of skill of everyday human object recognition. This project will address one gap in current knowledge, an understanding of the visual experiences that allow skilled object recognition to develop, by capturing and analyzing the visual experiences of 1- to 2-year-old toddlers. This is a key period for understanding human visual object recognition because it is the time when toddlers learn a large number of object categories, when they learn the names for those objects, and when they instrumentally act on and use objects as tools. Two-year-old children, unlike computer vision systems, rapidly learn to recognize many visual objects. This project seeks to understand how the training experiences (everyday object viewing) of toddlers may be optimal for building robust visual object recognition. The project aims to (1) understand the visual and statistical regularities in 1- to 2-year-old children's experiences of common objects (e.g., cups, chairs, trucks, dogs) and (2) determine whether a training regimen like that experienced by human toddlers supports visual object recognition by state-of-the art machine vision.

Considerable progress in understanding adult vision has been made by studying the visual statistics of "natural scenes." However, there is concern about possible artifacts in these scenes because they typically photographs taken by adults and thus potentially biased by the already developed mature visual system that holds the camera and frames the pictures. Also, photographed scenes differ systematically from the scenes sampled by people as they move about and act in the world. Accordingly, there is increased interest in egocentric views collected from body-worn cameras, the method used in the present work. Toddlers will wear lightweight head cameras as they go about their daily activities, allowing the investigators to capture the objects the toddlers see and the perspectives and contexts in which they see them. The research will analyze the frequency, views, visual properties, and range of seen objects for the first 100 object names normatively learned by young children, providing a first description of the early learning environment for human visual object recognition. These toddler-perspective scenes will be used as inputs to machine learning models to better understand how the visual information in the scenes supports and constrains the development of visual object recognition. Machine-learning experiments will determine which properties and statistical regularities are most critical for learning to recognize common object categories in multiple scene contexts. Data collected will be shared through Databrary, an open data library for developmental science.