Demba Ba - US grants

Deep learning is successful in practice because through large amounts of data and computation, useful general representations are learned that enable performance of complex tasks. Such representation learning is one of the most important and least understood aspects of deep learning — there are currently no quantitative measures for quality of representations, nor ways to certify that methods achieve the desired quality. This project is concerned with obtaining such measures and using both empirical and theoretical approaches to obtain certified representation-learning algorithms, as well as connecting these to representation learning in human and animal brains. Such an understanding is crucial for obtaining robust general algorithms that can be used for a wide variety of applications and tasks. This project will form new connections between machine learning, signal processing, statistics, and computational neuroscience. It will also result in stronger statistical guarantees for representation learning, placing it on a firmer mathematical foundation. As deep learning is used for increasingly consequential decisions, rigorous guarantees such as the ones pursued here become ever more important. Results of the project will also be used in education efforts at the K-12, college, and graduate level, including in programs aimed at groups historically under-represented in computing.

This project combines insights from machine learning, statistics, signal processing, and computational neuroscience to obtain a theory of representations in both artificial and natural neural networks. Specifically, the project aims to develop both task-dependent and task-independent measures of representation quality. Task-dependent measures capture the quality of representation through its performance in down-stream tasks, while task-independent measures define quality in terms of intrinsic properties of the representation and input distribution. The project will obtain relations between the two types and hence characterize conditions under which representation-learning algorithms transfer. The project will also result in rigorous bounds on representation quality under assumptions. Through the study of representation quality, the project will aim to explain prevalent features in real-world natural and artificial neural networks. These features include: locality in parameter space of neural responses in the visual and auditory system, mixed-selectivity of neurons that respond to signals of different types (for example, olfactory neurons in mice that respond to both spatial and odorant changes), and cross-modal neurons that respond to the same concept in signals of different types (for example, visual and auditory signals in the brain). The project will also connect representation learning to classical notions in signal processing and learning such as dictionary learning, as well as to well-known open questions in deep learning, including the prevalence of hierarchical representations, generalization of over-parameterized models, and simplicity bias.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.