Alice O'Toole - US grants

With ease human observers can recognize and identify familiar faces as well as extract additional information from both familiar and unfamiliar faces, including the sex, approximate age, race, and current emotional state of the person. Nevertheless, faces pose challenging computational problems for the perceiver. They are highly similar to one another, containing the same features arranged in roughly the same configuration. Perceivers must, therefore, be able to encode very subtle variations in the form and configuration of facial features. We develop a quantifiable theory of the perceptual information in faces and model the learning of this information. Faces are represented using "features" derived from the statistical structure of a set of learned faces, and the information most useful for discriminating among faces emerges as an optimal code. Our theory is implemented as a computational autoassociative memory (computer simulation) that operates on image-based codings of faces. The memory represents faces as a weighted sum of the eigenvectors (principal components, "features") of a covariance matrix of learned face images; these facial features may be displayed visually and are useful for both face recognition and visually-derived semantic categorizations of faces. We believe many face processing tasks and empirical phenomena are constrained more by perceptual factors than by complicated cognitive and semantic ones. Hence, our primary goal is to determine the extent to which perceptual constraints alone can account for these tasks and phenomena. As it is beyond the scope of the present proposal to examine all such phenomena, we have chosen a diverse subset. Our strategy in each case will be (a) to relate model-predicted accuracy and facial characteristic ratings to human measures of the same at the level of individual faces and (b) to alter face images synthetically so as to alter accuracy or ratings in predictable ways for human observers viewing the same set of faces processed by the autoassociative memory. We will address three issues: (a) typicality --more typical faces are less well recognized; (b) the perception of the sex of faces -- we model the structural differences between male and female faces and relate them to human ratings/performance using sex-linked facial characteristics; (c) the quantification and perception of the age of a face. Finally, we will analyze the eigenvectors in basic visual processing terms and compare the quality of face representations that emerge from principal components analysis as a function of spatial scale.

The human visual system can recognize a familiar face across wide variations of viewpoint, illumination, expression, and appearance. This remarkable computational feat is accomplished by large-­scale networks of neurons. We will test a face space theory of the representations that emerge at the top layer of deep learning convolutional neural networks (DCNNs) as a model of human visual representations of faces. Computer-based face recognition has improved in recent years due to DCNNs and the easy availability of labeled training data (faces and identities) from the web. Inspired by the primate visual system, DCNNs are feed­forward artificial neural networks that can map images of faces into representations that support recognition over widely variable images. Although the calculations executed by the simulated neurons are simple, enormous numbers of computations are used to convert an image into a representation. The end result of this processing is a highly compact representation of a face that retains image detail in an invariant, identity­-specific face code. This code is fundamentally different than any representation of faces considered in vision science. This theory we test combines key components of previous face space models (similarity, learning history) with new features (imaging conditions, personal face history) in a unitary space that represents both identity and facial appearance across variable images. We will test whether this model can account for human recognition of familiar faces, which is highly robust to image variability (pose, illumination, expression). The model will also be applied to understanding long standing difficulties humans (and machines) have with faces of other races. We aim to bridge critical gaps in our knowledge of how DCNNs work, linking psychological, neural, and computational perspectives. A fundamentally new theory of face representation will alter the questions we ask about face representations in all three fields. A new focus on understanding how we (or neural networks) ?perceive? a single familiar identity in widely variable images will give rise to a search for representations that gracefully merge the properties of faces with the real-­world image conditions in which they are experienced. This project presents a unique opportunity to study, manipulate, and learn from these representations, and to apply the findings to broader questions about high-­level vision from neural and perceptual perspectives.