Michael Black, Ph.D. - US grants

EIA- 0113679
Black, Michael
Brown University

ITR/SY: The Computer Science of Biologically-Embedded Systems

Biologically-embedded systems that directly couple artificial computational devices with neural systems are emerging as a new area of information technology research. The physical structure and adaptability of the human brain make these biologically-embedded systems quite different from computational systems typically studied in Computer Science.

Fundamentally, biologically-embedded systems must make inferences about the behavior of a biological system based on measurements of neural activity that are indirect, ambiguous, and uncertain. Moreover these systems must adapt to short- and long-term changes in neural activity of the brain. These problems are addressed by a multi-disciplinary team in the context of developing a robot arm that is controlled by simultaneous recordings from neurons in the motor cortex of awake behaving monkeys. The goal is to probabilistically model the behavior of these neurons as a function of arm motion and then reconstruct continuous arm trajectories based on the neural activity. To do so, the project will exploit mathematical and computational techniques from computer vision, image processing, and machine learning.

This work will enhance scientific knowledge about how to design and build new types of hybrid human/computer systems, will explore new devices to assist the severely disabled, will address the information technology questions raised by these biologically-embedded systems, and will contribute to the understanding of neural coding.

Current technology allows the simultaneous recording from hundreds of cells in the brains of awake behaving animals. Using this information to understand how the brain represents and processes complex information is a fundamental scientific challenge with large practical benefits to human health. In particular this project will develop technologies for neural prostheses which promise a new generation of therapies for the severely disabled to allow them to regain the ability to interact with the world. To that end, the project has three specific aims: 1) New probabilistic models of the neural code will be developed that exploit machine learning methods (Gibbs learning and boosting) and high performance computing resources. These models will represent the high dimensional probabilistic relationship between multiple behavioral variables and the firing activity of a population of neurons. 2) Neural decoding methods will be developed that model the uncertainty in neural recordings to make sound inferences that can drive neural prostheses. As part of this effort new probabilistic spike sorting algorithms will be developed and tested. 3) Adaptation of cells in the brain will be studied using the statistical models developed here and an understanding of this adaptation will be used to design new algorithms for prosthetic applications. These algorithms will themselves be adaptive in a way that optimizes prosthetic control in the face of a changing neural code. To learn such models from vast amounts of neural data, a new class of mathematical and computational tools is required. An interdisciplinary team including computer scientists, applied mathematicians, and neuroscientists will work in close concert to exploit existing infrastructure and experience with neural prostheses to address fundamental problems necessary for the practical application of these devices in humans. The coupling between the prosthesis application and basic research on neural coding provides a tight cycle of hypothesis, development, testing, and validation that will impact public health.

Robust intelligence rests on the ability to reason about missing, incomplete, ambiguous and corrupted data. This is particularly true in visual perception, where an intelligent system is faced with reasoning about the complexity of a changing three-dimensional world given only two-dimensional images. Bayesian inference has become popular for dealing with such problems because it provides a sound way of combining ambiguous sensor measurements with prior knowledge about the world. Priors represent the collected experience of a perceptual system and by integrating heterogeneous sources of information in a statistically sound way enable such a system to respond robustly to novel situations.

Markov random fields (MRFs) provide a powerful and popular formalism for representing visual priors. However, they have typically modeled only local, pairwise pixel interactions, which limit their modeling capabilities. This project aims at increasing the power and applicability of these models using larger pixel neighborhoods (cliques). The proposed Fields-of-Experts (FoE) model generalizes many previous MRF models, and all its parameters can be learned from real-world training data. Preliminary experiments have shown that, for example, image reconstruction applications benefit from such richer visual priors, but many other application domains have remained unexplored. The development of these statistical modeling tools will also have an impact on other domains outside of machine vision where the need for modeling complex, high-dimensional data arises. Finally, the dissemination of the collected experimental data, learned models, and software promises to stimulate research and make possible quantitative comparisons towards better statistical models of the visual world.

Proposal 0534858
"Statistical Models of the Primate Neocortex: Implementation and Application"
PI: Thomas L. Dean
Brown University

ABSTRACT

Theoretical accounts of the primate cerebral cortex (neocortex) are sufficiently rich in their predictive power and detailed in their specification that they warrant a concerted effort to implement and subject to computational experiment. This project applies recent work in theoretical neuroscience to develop statistical models and related learning and inference algorithms that capture the structure, scale and power of the neocortex for applications requiring robust associative recall, sensor fusion, pattern completion and sequence prediction. The cortical models and algorithms are implemented on moderate-sized computing clusters by distributing the computations among a large number of weakly coupled processes, each of which is capable of reproducing the aggregate behavior of a columnar cortical structure consisting of several thousand neurons. These simulated cortical columns are organized hierarchically much as they are in the primate neocortex. In addition to providing insight into the structure and function of the neocortex, the resulting algorithms and statistical models will enable researchers to combine lessons learned from biology with state-of-the-art graphical-model and machine-learning techniques to design hybrid systems that combine the best of biological and traditional computing approaches.

Modern experimental techniques in many fields of neuroscience can produce large quantities of data that can be processed and modeled in many ways, often providing the opportunity to answer questions beyond the original experimental motivation. Additionally, more and more sophisticated algorithms are being developed to analyze large neural data sets. An infrastructure that allows for routine sharing of data and algorithms will help advance the field while facilitating the validation of published results. Data sharing is now commonplace in other fields such as genomics and astrophysics and in these fields has accelerated the pace of research. There are numerous challenges (technical and social) involved in setting up and maintaining an infrastructure for sharing data and tools. The purpose of this meeting is to bring together a core group of experimentalists and modelers to examine ways in which such a system could be structured so as to best benefit the scientific community as well as the individual investigator.

This U.S.-Uruguay workshop award to Michael J. Black of Brown University will support the travel of 18 senior and junior researchers from US institutions to a workshop on "Vision in Brains and Machines" to be held in Montevideo Uruguay in November, 2006. Uruguayan organizers are Angel Caputi, a sensory neurophysiologist at the Clemente Estable Institute for Biological Investigations, and Gregory Randall, an expert in image analysis at the University of the Republic, both in Montevideo. The workshop will bring together researchers in the areas of neuroscience applied to human vision and of computer vision to explore the current state of the art in both fields and to find unexplored connections and collaborations. The workshop will combine background courses for students, lectures by leading researchers, and focused discussion groups exploring how each field and can learn from and leverage the other's results. New research directions resulting from roundtable discussions will be synthesized and broadly distributed.

The activity of synthesizing biological and machine perspectives on perception has benefits not just for the researchers but also for society in terms of potentially new treatments for vision loss (e.g. prostheses) and new machine-vision-enabled technologies. This workshop will enable South American researchers and students to interact with internationally recognized leaders in human neuroscience and machine vision and should foster interactions with senior and junior US researchers that could result in future research collaborations. This award is supported by the Americas Program of the Office of International Science and Engineering, and the Robust Intelligence Cluster of the Division of Information and Intelligent Systems.

This project will develop new approaches for recovering the three-dimensional (3D) shape and pose of the human body in images and video sequences. The methods will use a detailed 3D body model learned from laser range scans of over 2000 people. The approach will model the shape variation across people as well as the non-rigid shape variation due to changes in pose. The project will develop and test methods for robustly recovering the body shape in surveillance video sequences, in scenes with strong lighting, from collections of snapshots and in unconstrained television/film sequences. The recovered body model will be used to produce a variety of biometric measurements.

The majority of images and video sequences are of humans and recognizing people and their actions is a core problem in computer vision. The problem is challenging however because the human body is a complex, non-rigid, and articulated structure that can vary dramatically in pose, shape and appearance. Current methods focus on estimating human pose and typically ignore the problem of human shape estimation. This project will treat these problems together resulting in more robust solutions which will have a wide ranging impact in multiple disciplines. Human pose estimation is currently used in areas such as gait analysis, special effects, game development, human factors, and sports training to name a few. Robust video-based systems like the one developed here will extend the range of applications to home entertainment, elder care, autonomous vehicles and animal movement analysis. By extending previous methods to also estimate the three-dimensional shape of the human body in images and video sequences this project will enable additional applications in video forensics, surveillance, preventative medicine and special effects. More generally, methods like those developed here, that robustly recover the shape and pose of people in complex images and video streams, will be applicable to a wider range of problems in object recognition and tracking.

Project website: http://www.cs.brown.edu/~black/SCAPE.html

As we move through our visual environment, the pattern of light that enters our eyes is strongly shaped by the properties of objects within the environment, their motion relative to each other, and our own motion relative to the external world. This collaborative project will quantify motion within natural scenes, record activity from populations of neurons in the early visual pathway in response to the motion, and develop models of motion representation across neuronal populations. The primary goals of the work are to fully characterize the biological representation of motion in natural scenes in the early stages of visual processing that sets the stage for cortical computation critical for visual perception, and to unify the biological findings with computational models of motion from the computer vision community.

The perception of visual motion is critical for both biological and computer vision systems. Motion reveals structure of the world including the relative and absolute depths of objects, surface boundaries between objects and information about ego-motion and the independent motion of other objects. The effects of visual motion on the relationship between spatially localized and global properties of the natural visual scene, and how this is represented by the early visual pathway of the brain, are largely unknown.

This project addresses the computation of local and global properties of natural visual scenes by both distributed neural systems and computer vision algorithms using a novel set of complex naturalistic stimuli in which ground truth properties of the scene are known, and all aspects of the scene, including its reflectance, surface properties, lighting and motion are under investigator control. A unified probabilistic modeling framework will be adopted, that ties together the computational and biological models of properties of the natural scene. Neural activity will be recorded from a large population of densely sampled single neurons from the visual thalamus. From the perspective of the computer vision community, an important challenge exists in inferring the motion of the external environment (or "optical flow") from sequences of 2D images. From the perspective of the neuroscience community, quantifying the distributed neural representation of luminance and motion in the early visual pathway will be a critical step in understanding how scene information is extracted and prepared for processing in higher visual centers. A team of investigators with experience in computer science, engineering, and neuroscience will develop a theoretical foundation and rich set of methods for the representation and recovery of local luminance, local motion boundaries and global motion by brains and machines.

PROJECT SUMMARY / ABSTRACT Our overarching goal is to establish an animal model of freely moving humans. We choose to do so in order to directly measure the context-dependency of motor cortical activity and, ultimately, other activity reliant upon free movement such as social interaction among animals. Achieving this major technological challenge requires a complete system that includes (Specific Aim 1) wireless transmission of neural data from electrode arrays chronically implanted in monkeys, (Specific Aim 2) computer-vision algorithms to automatically extract body and limb orientation during free movement, and (Specific Aim 3) new mathematical and computational models to represent and extract information from high-dimensional neural and behavioral activity. This technology will enable an animal model of freely moving humans that will advance the development of cortical neural prostheses by providing models of the context-dependant nature of motor cortical control. Unlike traditional laboratory environments used to study animal movement, human amputees and tetraplegics operate in a variety of contexts that involve their movement in the world. Understanding the motor control of complex movement in these natural settings is absolutely critical for future advances in cortically-controlled prostheses. Given our overarching goal, our hypothesis is that motor cortical activity (e.g., directional tuning curves, absolute firing rates, correlations among units, etc.) will be different in important ways when rhesus monkeys perform the same reaching arm movements in an un-constrained context (e.g., not sitting quietly, not head restrained, not in dark and quiet room, etc.) as in a traditional, highly constrained context. Our three Specific Aims will put in place the electronic, computational and mathematical technology necessary to address this hypothesis, and also to make such studies of free behavior in rhesus monkeys possible. The innovative integration of neural engineering, neuroscience, computer vision, mathematics and neural modeling will provide new tools to enable the unprecedented study of motor control during natural, unconstrained behavior.

Computer Science (31)

Computer architectures have evolved greatly in recent years but these innovations are not widely taught in introductory computer organization courses. Existing simulators are used by researchers but do not meet the needs of students. The simulator developed for this project models a full x86 machine and includes modern architectural components including multicore, superscalar, two-level cache and speculation. Students select components such as the pipeline or memory to visualize how data are processed. Multiple levels of abstractions allow student to ignore components that they are not ready to study. One can also reconfigure the architecture components using a graphical user interface.

Expected outcomes include the simulator as a standalone application, a web applet and a collection of laboratory exercises. Materials are designed to be used within existing computer organization and architecture courses. The applet can be accessed by any modern browser and thus allows adoption by a wide audience. Pilot testing is being performed at American University, a small liberal arts school, and the University of Maryland, a large public university.

The investigators are developing an integrated, multidisciplinary undergraduate molecular forensics curriculum that fosters an experimental mindset, providing a platform to transform the way students experience the application of technology in scientific discovery. Combining analytical procedures in microbiology, biochemistry, and bioinformatics with algorithmic and computational skills in computer science, the program is engaging students across the College of Science and Mathematics and the College of Engineering in educational experiences involving cutting-edge applied research through the lens of an initial focus: production of a working database of E. coli fingerprints for use in tracking environmental contamination.

The curriculum is being delivered to over 1,000 students each year in multiple inquiry-based courses. These courses lay the foundation for a new emphasis in molecular forensics at Cal Poly, which leads students from the field, where E. coli is collected, to the laboratory, where the bacterial DNA is amplified and sequenced, to the computer, where the data are evaluated and archived in a database. Each lab module engages undergraduates in research with broad applications in the areas of food and water safety. Molecular forensics modules are integrated into the core of introductory courses at the Foundation level (freshman/sophomore), linking microbiology, biochemistry, and bioinformatics. At the Integration level (junior/senior), focused courses guide students in the construction and use of databases in molecular forensics. Two bioinformatics courses, one for life sciences majors and one for computer science majors, are being taught in concert to provide students with project-based experiences in multidisciplinary team settings. The student experience culminates at the Application level (senior project capstone) with teams of students working on research with environmental and health applications using the forensic database created in the program.

The project is cultivating an undergraduate research community while contributing to the field of molecular forensics. As diseases caused by food-and-water-borne bacterial pathogens continue to be a major public health issue, the importance of developing efficient methods for tracking the sources of outbreaks cannot be overemphasized. Throughout the molecular forensics curriculum, students are contributing to the development of solutions for rapid identification of contamination sources in recreational and potable water as well as food supplies. Computer science students are gaining the real-world experience of working directly with clients in the development of science-specific software applications.