Javier R. Movellan - US grants

Automatic Analysis of Spontaneous Facial Expressions
Abstract
The goal of this project is to develop computer systems for automatic analysis of spontaneous facial expressions, with a focus on the scientific study of the role of facial expressions in deception. A state-of-the-art digital video database of spontaneous facial expressions will be developed. This database will be hand-coded by behavioral scientist experts on facial expressions. This database will be used to develop an array of software tools for automatic analysis of facial expressions from video sequences. These tools will be developed by machine perception scientists in close collaboration with behavioral scientists and will be evaluated and refined for application to the scientific study of facial expressions.

The machine perception community is in critical need for standard video databases to train and evaluate systems for automatic recognition of facial expressions. This project will provide one such database and thus could potentially accelerate research in this field. Automated recognition systems would have a tremendous impact on basic research by making facial expression measurement more accessible as a behavioral measure, providing data on the dynamics of facial behavior at resolutions that was previously unavailable. Such systems would also lay the foundations for computers that can understand this critical aspect of human communication.

The goal of this project is to investigate a new class of stochastic probabilistic models, that make inferences and extract high-order dependencies from high-dimensional data. This research builds upon work that recasts existing unsupervised algorithms such as factor analysis, principle component analysis, and independent component analysis as energy-based feedback models. The diffusion network perspective investigated in this project affords new nonlinear generalizations of existing models, generalizations which are easily extended to overcomplete and undercomplete cases, and in which inference is relatively simple. Learning in diffusion based approaches is efficient and neurally plausible, making them good candidates for engineering applications and for modeling of neural and cognitive processes.

The new models will be explored using a synergistic combination of theoretical work and computer simulations. Their practicality will be tested by applying them to a variety of exciting and difficult machine perception tasks, including efficient representation of natural scenes, person identification and facial expression recognition, and reconstructing three-dimensionally transformed faces from 2-D video.

Because of their unique capabilities to model high-dimensional data in a nonlinear fashion, the diffusion networks studied in this project are likely to be important in a diverse group of fields including data mining, computer vision, and computational neuroscience, and could outperform existing algorithms in a wide variety of applications, including human computer interaction, image processing, and law enforcement.

The goal of this project is to develop systems that autonomously learn
to detect and track objects in video images. Current approaches
require large training datasets for which the object of interest has
been manually segmented by human operators. This process is very
laborious and time consuming, greatly limiting progress in the
field. This is arguably the main reason why computer vision technology
has not found a niche yet in every-day life applications. In this
project new machine learning and machine perception systems are being
explored that avoid the manual segmentation step. The training input
to these systems is a video dataset of unlabeled, naturally moving
faces in various background conditions. The target output is a state
of the art face detection system. The approach being explored is
based on the idea that one can develop sophisticated object detectors
in an unsupervised manner by biasing the training process using an
ensemble of low-level interest operators (motion, color, contrast).

This project is expected to provide a new class of machine learning
and machine perception algorithms that train themselves by observation
of vast image datasets. The use of large datasets is expected to make
a critical difference on the robustness of the systems and to allow
them to handle realistic every-day life environments. Such systems
would have significant applications in education, security, and
personal robots. Besides its practical applications, this project has
the potential for broad scientific implications in machine learning,
machine perception, and developmental psychology.

Abstract
Program: NSF 04-588 CISE Computing Research Infrastructure
Title: CRI: COLLABORATIVE RESEARCH: Automated facial expression measurement: toolbox and database

Lead Proposal: CNS 0454233
PI: Marian S. Bartlett
Institution: University of California-San Diego

Proposal CNS 0454183
PI: Mark Frank
Institution: Rutgers University New Brunswick


This community resource project will package and release to the CISE research community a collection of software tools for vision based perceptual primitives for human-computer interaction studies and a database of facial expressions that has been coded by facial expression experts. The tools enable recognition of basic emotions and facial actions from the Facial Action Coding Systems (FACS). The tools and database have been developed with experts from machine learning and facial expression fields. Broader impacts of this community resource include enabling research and education in this area for a broader community in the U.S., use of automated facial expression in education and machine tutoring applications, enabling advances in computer vision, enabling advances in behavioral science and medicine related to emotion, cognition, and human-machine communication, and applications in homeland security.

Proposal Number: 0622229

Proposal Title: Infomax Neural Networks for Real-Time Learning and Control

PI Name: Movellan, Javier R.

PI Institution: University of California-San Diego

The objective of this project is to develop principled methods for
solving Infomax Control problems, i.e., problems that require the
control of active sensors so as to maximize the value of the
information they gather.


Intellectual Merits:

Recent advances in stochastic optimal control, neural networks, and
machine learning have made it possible to approach this problem in a
principled manner. The proposed approach relies on three
methodologies: (1) Mathematical modeling; (2) Empirical studies to
evaluate the predictions made by the models; (3) Development and
evaluation of robotic systems.


Broader Impacts:

Infomax control is a critical problem with applications to robotics,
autonomous navigation, distributed control, automatic teaching
systems, and national defense. For example, the approach can be
applied to develop tutoring automatic tutoring systems that ask
questions so as to optimize the material learned by the students. It
can also be used to develop algorithms for data-mining large datasets
so as to acquire the relevant information as quickly and efficiently
as possible. It can also be used for developing robots that actively
scan the environment gathering the information that is most important
to achieve their current goals. Finally the approach will provide a
mathematical foundation to help understand how the brain connects
perception and action.


In addition to the funding of graduate students, research and
education activities will be developed in conjunction with the
Engineering Course at the Preuss School in San Diego, a
charter School for low-income student in grades 6-12. The proposed activities will serve as an example of an approach to introduce and teach the latest technological and scientific research in local middle and high-schools.

This project is exploring the integration of video and multiscale visualization facilities with computer vision techniques to create a flexible open framework to advance analysis of time-based records of human activity. The goals are to (1) accelerate analysis by employing vision-based pattern recognition capabilities to pre-segment and tag data records, (2) increase analysis power by visualizing multimodal activity and macro-micro relationships, and coordinating analysis and annotation across multiple scales, and (3) facilitate shared use of the developing framework with collaborators. Researchers from many disciplines are taking advantage of increasingly inexpensive digital video and storage facilities to assemble extensive data collections of human activity captured in real-world settings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. The main obstacles to fully capitalizing on this scientific opportunity are the huge time investment required for analysis using current methods and understanding how to coordinate analyses focused at different scales so as to profit fully from the theoretical perspectives of multiple disciplines. Thus, any research using video or other time-based records in order to document or better understand human activity is a potential beneficiary of this research, and the long range objective is to better understand the dynamics of human activity as a scientific foundation for design.

The goal of this project is to make progress on computational problems that elude the most sophisticated computers and Artificial Intelligence approaches but that infants solve seamlessly during their first year of life. To this end we will develop a robot whose sensors and actuators approximate the levels of complexity of human infants. The goal is for this robot to learn and develop autonomously a key set of sensory-motor and communicative skills typical of 1-year-old infants. The project will be grounded in developmental research with human infants, using motion capture and computer vision technology to characterize the statistics of early physical and social interaction. An important goal of this project is to foster the conceptual shifts needed to rigorously think, explore, and formalize intelligent architectures that learn and develop autonomously by interaction with the physical and social worlds. The project may also open new avenues to the computational study of infant development and potentially offer new clues for the understanding of developmental disorders such as autism and Williams syndrome.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5). The goal of this project is to develop computational models of the nonverbal behavior and interactive strategies observed during face-to-face teaching. These computational models will serve as a foundation for a new generation of embodied teaching agents that approximate the benefits of face-to-face human tutoring. The project will help advance the science of learning and teaching by improving our understanding of the dynamics of nonverbal behavior in teaching at a computational level, across multiple time scales: From low-level micro-expressions in the timescale of tens of milliseconds, to cognitive and affective processes with time scales of seconds, to higher level strategic behaviors operating at longer time scales.

In addition to its scientific and technological value, this project has a significant outreach component. The project would help grow links between a research oriented campus (UCSD) an undergraduate teaching university (SDSU). The robotics aspects of the project will be developed in collaboration with the Preuss School Robotics Club. The Preuss School is a charter school for low-income students in grades 6-12 and is currently ranked as one of the top high schools in the nation.

The NSF-funded research project led by Javier Movellan at the University of California, San Diego will develop a social network (?RubiNet?) for early childhood education. The network will help integrate the activities of children, teachers, parents, and researchers with the goal of improving early childhood education at a national and international level. RubiNet will use low-cost sociable robots to connect children and teachers in different classrooms, from different socioeconomic backgrounds, cultures, and ethnicities. Some of the classrooms will be in different countries. Movellan and his colleagues will investigate the potential of RubiNet as a tool for rapid design and evaluation of low cost early intervention programs. The potential for RubiNet to improve academic skills in young children will be assessed. The researchers will also investigate whether this form of interaction engenders more familiarity and positivity in children's views of those who are different from themselves.

An unprecedented number of children in the US start public school with major deficits in basic skills, including social skills, language, and mathematics. Children that experience early failure in school are more likely to later become inattentive, disruptive, or disengaged. These students tend to drop out of school early, and are more likely to depend on public assistance programs for survival. Empirical research suggests that the key to avoiding this vicious cycle is to intervene during the pre-kindergarten years instead of waiting until failures occur in kindergarten or later. Unfortunately, early intervention programs are typically very costly. Thus, another key goal of RubiNet is to significantly lower the cost of intervention in early education. In addition, RubiNet will be of great potential benefit to researchers who study social interaction in young children.

It is commonly accepted that there is a crisis in education in the US. There are too many struggling learners, too many students who cannot read or do basic arithmetic, let alone advanced mathematics. What is not commonly accepted is what to do about this crisis. The researchers at the Temporal Dynamics of Learning Center (TDLC) believe that part of the current crisis in education is the lack of scientific understanding of how the brain learns, and the lack of translation of this scientific understanding to the classroom. An essential, yet understudied, component of learning that could have a strong impact on education is the role of time and timing in learning. TDLC brought together an interdisciplinary team of over 40 investigators from 16 different research institutions in order to focus research energy on this goal. TDLC's purpose is to achieve an integrated understanding of the role of time and timing in learning, across multiple time and spatial scales, brain systems, and social systems, to 1) create a new science of the temporal dynamics of learning; 2) to use this understanding to transform educational practice; and 3) to create a new collaborative research structure, the network of research networks, to transform the practice of science.

Why study timing? Timing is critical for learning at every level, from learning the precise temporal patterns of speech sounds, to learning when to give feedback in the classroom, to the optimal frequency and timing of studying new material. Moreover, a decade of neuroscience research demonstrates that the intrinsic temporal dynamics of the brain itself also reinforce and constrain learning. For example, work at TDLC has shown that measurements of the brain waves of a toddler-the temporal dynamics of thought - can predict how well that child will perform at language tasks years later. This provides the possibility that early intervention could overcome these difficulties, demonstrating the usefulness of studying temporal dynamics. A research program of this size and scope is clearly only possible through the Center Funding model, in order to provide resources at the scale necessary to coordinate the large team of researchers. The work is organized by dividing the personnel into four research networks, where researchers from multiple disciplines are interested in common questions, and who synchronize their research around experiments that can be carried out in humans, animals, and computational models, allowing unprecedented convergence of techniques on a single question.