Tai Sing Lee - US grants

This project is being funded through the Learning and Intelligent Systems (LIS) initiative. It is concerned with learning that is incremental in nature, resulting in skills that improve over time. Such learning is responsible for the development of perceptual discriminations, stimulus associations, and motor skills, and is seen in both humans and animals. The goal of the research is a systems-level neural theory of incremental learning. The project combines computer simulations with neurophysiological recording from the brains of behaving rats and monkeys, functional magnetic resonance imaging in humans, and robotic implementation. It is known that many parts of the mammalian brain contribute to incremental learning. Although the cortex may play a central role, other brain areas known to make vital contributions include the basal ganglia, hippocampus, amygdala, and cerebellum. It is important to understand the roles of these various areas and their interactions with each other during learning. Experimental work to be performed includes recording from various areas of the brains of rats and monkeys during learning and imaging the brains of humans performing analogous tasks. Models of neural function will be developed based on the monkey experiments and will be implemented on a mobile robot. These implemented models will allow the training of the robot, and a comparison of the learning that takes place with that observed in humans and animals. The development of theories of incremental learning will provide a better understanding of how the process occurs and may result in improved approaches to the development of skills in both humans and machines.

This award supports the study of computational principles and neural mechanisms underlying active perception -- the active gathering of information to construct, update and improve a system's representation,knowledge and understanding of its environment. The PI will develop an integrated multidisciplinary research and educational program that utilizes techniques from computer and information science, system engineering, neuroscience and psychology to study how monkeys and humans use eye movement to actively analyze visual scenes in different visual tasks and how the neural machinery and representations are transformed in these processes. To prove the theoretical framework he is developing, he will build an adaptive active vision system that will learn to see. Along with the research, he will enhance the traditional computer science and computer vision curriculum at Carnegie Mellon by introducing problems,theories and practice from neuroscience and neural computation to computer science classrooms and laboratories to prepare a new generation of interdiscciplinary computer scientists for the new challenges in computational biology.

This is a multidisciplinary research program for investigating both the computational principles and the neural mechanisms underlying our visual perception of three-dimensional (3D) surfaces and shapes in the natural world. Its goal is to understand how surfaces of objects are inferred and represented in the brain. The general approach is first to discover the statistical regularities of patterns and structures in 3D natural scenes and to develop a computational framework for representing and inferring these structures from optical images; and second to test neurophysiologically the predictions generated by the computational framework on the neural basis of surface representation and inference. The fundamental hypothesis is that the visual system functions as a hierarchical probabilistic inference system in which the feedforward and feedback connections among the different visual areas in the cortical hierarchy serve to mediate two-way Bayesian belief propagation. In this framework, the brain is conjectured to actively construct a representation of the visual scene based on the retinal input as well as our prior knowledge and experience of the world. The investigator will carry out a novel statistical study of 3D natural scenes, develop efficient probabilistic computational algorithms for surface inference based on natural scene statistics, explore neural models for implementing such algorithms, and test neurophysiologically these models by recording and analyzing neuronal activity in the early visual areas of primate cerebral cortex. It is a tightly coupled interdisciplinary project that involves synergistic research in computer vision, computational neuroscience and systems neuroscience to address fundamental questions in these three fields. Understanding how the brain makes inference about the visual world will have a significant broad impact on neuroscience, clinical medicine and robotics. This integrated study of a hierarchical visual inference system and its associated probabilistic inference algorithms, rooted in natural scene statistics, will contribute to the foundation for building a new generation of flexible and intelligent robotic vision systems. Such systems will be able to learn and adapt to the statistical regularities of a changing environment and make inferences based on scene contexts. The proposed research program also provides an unique educational vehicle of interdisciplinary training to graduate and undergraduate students that will serve as a catalyst to integrate computer science research and biological research in the scientific community at large.

Spectral Graph Theory or Algebraic Graph Theory, as it is also known,
is the study of the relationship between the eigenvalues and
eigenvectors of graphs and their combinatorial properties. Random
walks on graphs, expander graphs, clustering, and several other
combinatorial aspects of graphs are intimately connected to their
spectral properties. Recent approaches to the analysis of
high-dimensional data have exploited the fundamental eigenvectors of
the data. These data sets are large and ever increasing requiring
``real-time" accurate responses to the given queries. This creates the
need for very fast algorithms, that also provide strict theoretical
guarantees on their output. Spectral techniques have been applied to image
processing, both by computers and in the primary visual cortex of
monkeys. Critical component to all these application is algorithms
with efficiency and accuracy guarantees for solving these linear system
and finding their fundamental eigenvectors.


A multidisciplinary team consisting of Theoretical Computer
Scientists, Machine Learning Scientist, and Neuroscientist will
develop and apply spectral graph theory to applications from data
mining to clustering, and image processing. Enabling technology
develop will include: 1) linear-work or O(m log m)-work algorithms
that run in poly-logarithmic parallel time for computing extreme
eigenvalues and generalized eigenvalues of diagonally-dominant
matrices, including Laplacian matrices, as well as algorithms of
similar complexity for solving the related linear systems. 2) Better
estimates for Fiedler values and generalized Fiedler values.
Application development: 1) Improvements in spectral image
segmentation. 2) The use of generalized eigenvalues in data mining and
image segmentation to combine multiple sources of information. 3) The
use of preconditioners for approximate inference in graphical models.
and 4) Combine insights into the problem of image segmentation gained
from spectral algorithms with knowledge gained from recent experiments in visual system
of monkeys to better understand how the primary visual cortex functions.

Computational and neurophysiological investigation of robust visual inference

Tai Sing Lee, Carnegie Mellon University

This project is a multi-disciplinary investigation of the computational principles and neural mechanisms underlying robust visual inference in primate systems and the exploitation of these principles to develop new statistics-based computer vision approaches for inferring 3D scene structures in visual images. An image of a real 3D scene is highly ambiguous and difficult to interpret because it could be generated by many possible combinations of the different physical causes, such as lighting, texture and shapes. Classical approaches in computer vision attempt 3D scene inference by modeling these image formation processes with simplified assumptions and then inverting these models. The PI proposes a statistical approach to better solve these problems by learning and exploiting the statistical priors on 3D shapes in the natural environment and their correlational structures with 2D images. The PI plans to develop efficient Bayesian belief propagation algorithms within the framework of probabilistic graphical models that allow flexible incorporation of rich statistical scene priors. The computational work will guide his investigation of the neural encoding of scene priors and the mechanisms of probabilistic inference in the primate early visual cortex using advanced electrophysiological techniques. A better understanding of the neural representations of priors and mechanisms of inference will represent a fundamental scientific advance in neuroscience and will also provide new insights for improving the statistics-based computational approaches for visual inference.

This interdisciplinary research project seeks to elucidate the computational machinery and algorithms in our brain that enables us to perceive 3-dimensional surfaces of objects in visual scenes based on the 2D images projected on our retinae. The first conjecture of the project is that the neural circuitry underlying perceptual inference in our brain can be predicted by the statistical structures in our natural environment. To prove this conjecture, statistical studies on 3D natural scenes will be carried out and their predictions on neural connectivity will be compared with the functional connectivity and tuning properties of depth-sensitive neurons in the primate visual cortex obtained using large-scale multi-electrode recording techniques. This will provide deeper insights into how the brain represents and builds models of the structures of the external world to enable perceptual inference. To understand what such circuits can compute, the investigator conjectures that neural circuits realize a class of generative models in computer vision and computational neuroscience called Markov random fields and Boltzmann machine. This second conjecture will be evaluated by exploring the theoretical link between the neural circuits and these computational models, by comparing experimental neural observations with behaviors of these computational models, and by evaluating the computational performance of the inferred neural circuits in solving stereo computation and surface interpolation problems in real world data. The research combines techniques in machine and statistical learning, computer vision, neural networks and neurophysiology to dissect neural circuits from a functional perspective. By linking real circuits to a powerful class of computational models in statistical inference, the project will have broader impacts by providing novel evidence and fundamental insight to the neural mechanisms and computational algorithms underlying statistical perceptual inference in the brain. This interdisciplinary project will be a catalyst for the development of educational initiatives to bring bringing computer science and neuroscience together for undergraduates and graduate students, and to promote the awareness and involvement of students from multiple disciplines, including under-represented groups, in the field of computational neuroscience.

In this interdisciplinary project, machine learning approaches are coupled with neurophysiological studies of primate early visual cortex to investigate the functional, coding and computational benefits of the observed neural representation and computing architecture. Neural models, with recurrent connections and the proposed dual-code strategy, will be developed to solve multiple vision problems simultaneously and to fit neurophysiological data. The representations will be studied from both coding perspectives and computational perspectives, based on scene statistics and their relevance for solving vision problems. The research program will be facilitated by international collaboration and tightly integrated with undergraduate and graduate education in neural computation. The proposed project wide provide new insights to the computations and functions of the biological visual system, as well as new ideas and inspirations for developing machine learning systems that can learn from limited data and function robustly and flexibly in novel complex situations, potentially with broad societal and technological impact.

Current deep learning neural networks utilize tens or hundreds of layers to learn solutions for specific computer vision problems. The mammalian visual system has much fewer layers, and yet can solve many tasks in a variety of novel and complex situations. The nervous system might achieve this feat by having neuronal circuits with loops and recurrent connections, and with order of magnitude more neurons in each "layer." Recent neurophysiological findings suggest that neurons in the primary visual cortex (V1) of primates are not simply oriented edge and bar detectors as described in textbooks, but respond strongly to highly specific complex local patterns, although they also respond to many other patterns with much weaker responses. The PI proposed that the individual neurons are not amorphous entities, functioning facelessly in a large population, but are distinct and unique individuals that serve as specialists for some specific tasks and as generalists in other tasks. They participate in population encoding of information with strong sparse codes or weak distributed codes respectively, depending on the functional roles they serve.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.