Emanuel Todorov - US grants

[unreadable] DESCRIPTION (provided by applicant): The goal of this project is to develop an easy-to-use data analysis system for fast, accurate, and robust estimation of the multijoint movement trajectories of the human body. Such technology has a number of medical applications, including clinical gait analysis, physical medicine and rehabilitation, sports medicine and injury prevention, quantitative assessment of motor dysfunction, design and fitting of orthoses and prostheses, feedback control of neuromuscular stimulators, calibration of implantable sensors. Access to reliable multijoint estimation tools is also a prerequisite for continued progress in basic motor control research on both humans and other species. Motion capture hardware has become widely available, and allows fast and reasonably accurate measurement of the position, orientation, bending, acceleration, etc. of various makers attached to the body. The available data analysis tools, however, lag behind these hardware advances; estimating the configuration of a multiarticulate body to which markers are non-rigidly attached remains a challenging problem. In particular, a) existing methods assume rigid marker attachment and provide no estimate of the errors resulting from unavoidable soft tissue deformation and miscalibration; b) placing markers at predetermined locations and measuring limb sizes for each subject requires prolonged setup sessions; c) the reliance on sensor-specific estimation methods makes it difficult to utilize new sensor modalities or placements; d) the redundancy in the sensor data due to the body structure is rarely exploited to handle missing data, marker misidentification, and noise in general; e) kinematic estimation is performed separate from dynamics and therefore can produce dynamically impossible trajectories; f) the few existing systems that utilize more general iterative minimization techniques do not guarantee real-time performance; g) most existing systems are tailored to the needs of the computer animation industry and do not even attempt to meet the accuracy requirements for research and clinical tools; h) investigators who need such tools are faced with the daunting task of developing their own. We propose to develop an integrated system that addresses all of the above problems. Our approach is based on a general probabilistic formulation, which allows us to apply a combination of modern statistical estimation, numerical optimization, and software engineering techniques. We believe that the multiple core methodologies needed to develop such a solution are already available, albeit in different literatures, and the time is ripe to bring them together. Our longterm goal is to provide a satisfying solution to the problem of marker-based multijoint estimation, as well as to incorporate the system proposed here into a larger suite of software tools for biomechanical analysis and simulation that is currently being developed at USC. The proposed system will not only be used in our own research, but will be documented and made available to other investigators interested in complex many-degree-of-freedom movements. [unreadable] [unreadable] [unreadable]

Proposal Number: ECS-0524761

Proposal Title: Hierarchical optimal control of complex dynamics - new algorithms and models of sensorimotor function

PI Name: Todorov, Emanuel

PI Institution: University of California-San Diego

Intellectual Merit: The PI plans to use and integrate more recent methods of adaptive dynamic programming or "reinforcement learning," and apply them to the modeling and control of biomechanical systems like human arm movement. Reinforcement learning methods have been applied before in biology and in the study of arm movement, but past studies have mainly relied on old, simple mathematical structures which do not scale well to high degrees of complexity in space and time. This project will make a unique effort to reach out, integrate and use more advanced methods. The effort to understand the mathematical, functional basis of effective decision and control in the brain is perhaps one of the most important, fundamental challenges before science in general.

Broader Benefits: Cross-disciplinary communication between the most advanced areas of technology and the serious study of intelligence in the brain is still far less than it could be. If successful, this project could have a major impact on the unification of knowledge across disciplines. Aggressive education and dissemination are a natural part of the effort to heal the gap between disciplines related to these scientific goals.
Better understanding of biomechanical issues may also have important benefits both in medicine and in robots.

[unreadable] DESCRIPTION (provided by applicant): We request support for the symposium Advances in Computational Motor Control, which is a one-day annual satellite event to the Society for Neuroscience meeting. It has already been running for five years with increasing popularity; current attendance is about 200 people per year. We ask for funding for the next five years. The symposium aims to attract the best work in sensorimotor control that has a theoretical component. While formal ideas expressed as computational models are preferred, intuitive ideas that await formalization are also welcome. We encourage presentations by the researchers who are most directly involved in the work being submitted. As a result the majority of speakers are graduate students and postdocs, including a substantial percentage of women and minorities. Contributed talks are selected through a rigorous and objective peer-review process. The acceptance rate is very low, about 30%, resulting in a program with exceptional quality. The program also features a couple of invited talks by prominent researchers working in motor control or related fields. Most talks include a mix of modeling and empirical work, facilitating the interaction between theorists and experimentalists. We request support for the symposium Advances in Computational Motor Control, which is a one-day annual satellite event to the Society for Neuroscience meeting. It has already been running for five years with increasing popularity; current attendance is about 200 people per year. We ask for funding for the next five years. The symposium aims to attract the best work in sensorimotor control that has a theoretical component. While formal ideas expressed as computational models are preferred, intuitive ideas that await formalization are also welcome. We encourage presentations by the researchers who are most directly involved in the work being submitted. As a result the majority of speakers are graduate students and postdocs, including a substantial percentage of women and minorities. Contributed talks are selected through a rigorous and objective peer-review process. The acceptance rate is very low, about 30%, resulting in a program with exceptional quality. The program also features a couple of invited talks by prominent researchers working in motor control or related fields. Most talks include a mix of modeling and empirical work, facilitating the interaction between theorists and experimentalists. [unreadable] [unreadable] [unreadable]

Proposal Number: 0702221

Proposal Title: Hierarchical approximations to optimal control

PI Name: Todorov, Emanuel

PI Institution: University of California-San Diego

The objective of this research is develop new algorithms for
approximately-optimal control of complex dynamical systems. The approach
combines inspiration from neuroscience with mathematical advances in control
theory. The algorithms have a hierarchical structure reminiscent of the way
the brain generates complex behavior. The lower level of the hierarchy
augments the body and makes it easier to control. The higher level monitors
progress and steers the system towards achievement of the common task. In
this way the complexities due to the body are separated from those due to
the task.

Intellectual merit
The project includes two complementary classes of algorithms. The first
class represents a significant advance in the theory of stochastic optimal
control. A general family of problems are identified where the fundamental
equations characterizing the optimal solution turn out to be linear, even
though the controlled system is nonlinear. The second class of algorithms
represents a practical framework for attacking high-dimensional nonlinear
problems, particularly those that arise in biomechanics.

Broader impacts
The proposed theoretical developments represent foundational work which is
likely to have a lasting impact. The proposed numerical algorithms have the
potential to extend the range of practically-solvable optimal control
problems. Optimal control is of interest in many fields of science and
engineering, including the recovery of motor function via brain-machine
interfaces. Educational activities include mentoring of the graduate
students funded by this proposal, as well as design and teaching of both
graduate and undergraduate classes at the interface of Neuroscience and
Engineering.

The objective of the proposal is to achieve a core competency in the field of computational mechanics at The University of Iowa through the development of an innovative method, referred to as the dimensional decomposition method, for solving a general random eigenvalue problem in modeling and simulation of stochastic dynamic systems. The proposed effort will be based on: (1) new decomposition method for lower-dimensional approximations of general complex-valued eigensolutions of random eigenvalue problems; (2) new multipoint decomposition and monomial preconditioner for probabilistic characteristics of eigensolutions; and (3) new design sensitivity formulation for analytic gradients of probabilistic measures of random eigensolutions. The proposed research is ambitious and novel, differing in fundamental ways from most prior research in this area. The methods to be developed will address highly nonlinear input-output transformations, an unlimited number of random variables or fields, and arbitrarily large uncertainty of random input. Due to innovative formulation of the analytically derived stochastic design sensitivities, subsequent optimization of dynamic systems can be conducted employing any standard gradient-based algorithm. The decomposition method will aid in solving large-scale, multidisciplinary, stochastic eigenvalue problems in engineering and science.

The proposed research will be of significant benefit to numerous commercial and industrial applications, such as civil, automotive, and aerospace infrastructure. Potential engineering applications include analysis and design of civil structures; noise-vibration-harshness of ground vehicle systems; fatigue durability of aerospace structures; and reliability of microelectronics and micro-electro-mechanical systems. Beyond engineering, potential applications include nuclear physics, number theory, computational biology, and computational finance, among others. Therefore, the research proposed here will positively impact a number of areas of national significance. The transfer and dissemination of knowledge created by this project will take place through continued collaboration with industries, organization of symposia in ASME conferences, journal publications, presentations and publications at major conferences and institutions, and student education. Partnerships with two government and industrial laboratories will enable implementation of the basic methods developed in this project to resolve several large-scale industrial problems. The educational goals comprise recruitment of a Ph. D. student from underrepresented minority or women groups, implementation of software tools from this project in upgrading courses in The University of Iowa's principal engineering programs, and authoring a research monograph.

This project aims to reverse-engineer the human brain's ability to control the hand.
The project begins by combining a robotic hand previously developed by the PI with a new type of sensitive skin, with a hundred biomimetic tactile sensors.

The main goal of this project is to understand how it is possible to achieve dextrous, approximately optimal control of a hand, performing familiar but challenging tasks in manipulating objects. New, more advanced learning-based control algorithms will be developed and tested on the four empirical testbeds of the project: (1) robotic manipulation by the biomimetic hand; (2) data from recording of human hands performing the same tasks; (3) computer simulations of physical hands; and (4) computer control of cadaver hands via their tendons. The project will use the same algorithms both as models of human motor control and to go beyond the present state of the art in robotic manipulation; this unified approach to biology and engineering is an essential part of the transformative goals of the COPN topic. Dextrous robotic hands have a wide variety of possible applications in industry, space and national security. Improved understanding of how humans can learn to perform better with their hands will also have broader benefits, particularly for the disabled. The team proposes a vigorous plan for education and outreach, capitalizing on the human interest aspects of the demonstrations they will be developing.

The goal of this project is to make progress on computational problems that elude the most sophisticated computers and Artificial Intelligence approaches but that infants solve seamlessly during their first year of life. To this end we will develop a robot whose sensors and actuators approximate the levels of complexity of human infants. The goal is for this robot to learn and develop autonomously a key set of sensory-motor and communicative skills typical of 1-year-old infants. The project will be grounded in developmental research with human infants, using motion capture and computer vision technology to characterize the statistics of early physical and social interaction. An important goal of this project is to foster the conceptual shifts needed to rigorously think, explore, and formalize intelligent architectures that learn and develop autonomously by interaction with the physical and social worlds. The project may also open new avenues to the computational study of infant development and potentially offer new clues for the understanding of developmental disorders such as autism and Williams syndrome.

de Sa
0756828

Brain computer interfaces (BCIs) translate basic mental commands into computer-mediated actions. BCIs allow the user to bypass the peripheral motor system and interact with the world directly through brain activity. These systems are being developed to aid users with motor deficits which can stem from: neurodegenerative disease (such as Lou Gehrig's disease, or ALS), injury (such as spinal cord injury), or even environmental restrictions which make movement difficult or impossible (such as astronauts in space suits). BCI systems typically require extensive user training to generate reproducible and distinct brain waves. Furthermore, until very recently, most BCI systems have interacted with the user in unintuitive or unnatural ways, such as moving a cursor or bar left and right by engaging in two unrelated forms of mental imagery, such as moving the right hand vs. the left foot. Realistic visual feedback of interpreted motor action should substantially improve usability and performance of BCI systems. This hypothesis is based on four observations: 1) humans have evolved to adapt their motor control in response to visual and proprioceptive feedback; 2) rapid motor adaptation is demonstrated in virtual reality experiments; 3) animals improve their neural signal when given visual feedback of their decoded neural activity; and 4) visual feedback of interpreted movement should activate the mirror neuron system, producing a stronger movement signal. The proposed work aims to improve upon current BCI systems based on motor imagery by providing more natural and lifelike feedback. This task can be broken down into 3 main objectives: 1) analyze motor imagery with visual feedback in an offline setting; 2) develop algorithms for real-time EEG analysis; and 3) construct a real-time BCI system utilizing lifelike motion animations as visual feedback. While results of objectives 1 and 2 should each in their own right contribute to the current state of the art in BCI systems, the largest BCI performance and usability gains should be made by introducing lifelike feedback into an online paradigm in the third objective. The proposed system can also be used to study learning and sensory-motor processing in normal subjects by studying their adaptation to the system. It may also inform more costly invasive recording experiments by helping to determine optimal placements of implants. All software written for EEG signal processing and analysis will be made available as add-ons to EEGLAB which is distributed in accordance with University of California policy for research, education, and non-profit purposes. The EEGLAB project is also developing an EEG database in conjunction with the San Diego Supercomputer Center. Representative data sets will be released via this database in accordance with University of California policy.

DESCRIPTION (provided by applicant): We propose to further develop our optimal feedback control theory of motor coordination, and utilize its potential to explain known phenomena and as well as novel experimental results on goal-directed arm movements in 2D and 3D. We will use the theory to shed new light on several important issues in sensorimotor control: regularities in kinematics and muscle activations, task-specific impedance and responses to perturbations, origins and structure of motor variability, and eye-hand coordination patterns. In addition to basic research, the project involves a substantial bio-engineering component with direct applications to health. We will construct detailed musculoskeletal models of the human arm and develop hierarchical control algorithms capable of making the model arm accomplish behavioral goals in real time. Such algorithms can then be used to control functional electric stimulators and robotic prostheses, and thereby restore motor function and improve the quality of life of disabled patients. PUBLIC HEALTH RELEVANCE: We will develop a general mathematical theory of how the brain controls arm movements. Better theoretical understanding of motor function can facilitate the design of new treatments for movement disorders. We will also develop automatic control algorithms that can use signals extracted from the brain to make a prosthetic arm accomplish desired movement goals.

DESCRIPTION (provided by applicant): We propose to develop a toolbox for estimation, simulation and control of multi-joint movements. Our immediate goal is to facilitate research in Motor Control, by providing access to advanced computational methods and making such methods an integral part of the hypothesis generation-and-testing cycle. The estimation component of the toolbox will enable researchers to accurately compute multi-joint movement trajectories as well as limb sizes from motion capture data, without spending hours to place markers at precise locations and redesign setups to ensure that every marker is always visible. The control component will make it possible to formulate mathematically-sound hypotheses about the control strategies used by the brain, and automatically synthesize detailed control laws corresponding to the user's hypotheses. These control laws will then be applied to realistic musculo-skeletal models, using the simulation component, and the predicted behavior will be compared to experimental data in terms of kinematics, contact forces and EMGs. In case of a mismatch the toolbox will be able to netune any free parameters of the controller, and also search the library of candidate control strategies and identify the one which best agrees with the data. Our longer-term goal is to assist clinicians and engineers designing new treatments such as reconstructive surgery and functional electric stimulation. Testing candidate control mechanisms on simulated musculo-skeletal dynamics can greatly reduce the undesirable trial-and-error iterations. Customizations necessary for clinical use are left outside the scope of this project, however they will be possible once the core functionality is developed in a system with open design. The toolbox will be written in Matlab, with some C++ components, and will be freely available for academic, research and non-prot purposes. Project narrative We propose to develop a toolbox for estimation, simulation and control of multi-joint movement. Our immediate goal is to facilitate research in the eld of Motor Control by providing access to advanced computational methods presently beyond the reach of many investigators. While tools for simulating musculo-skeletal dynamics already exist, simulation alone is rarely suffcient to advance our understanding of motor function. Here we will combine simulation with automatic controllers capable of driving realistic musculo-skeletal models, and provide tools for estimating multi-joint movements from motion capture data. Our longer-term goal is to assist clinicians and engineers designing new treatments such as reconstructive surgery and functional electric stimulation.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

0930927
Matsuoka

About 700,000 Americans suffer from new or recurrent strokes each year, with some 500,000 being fortunate enough to survive [1]. Over half of all strokes occur in areas of the brain that control movement, leaving many victims with impaired motor functions [2] that are most often localized in one side of the body and in one specific area, such as the arm or hand.

With careful physical rehabilitation, damaged motor functions can recover partial or full mobility as the nervous system rewires its neural circuits to represent lost functions at new neural locations. The level of recovery depends on the amount and quality of post-stroke rehabilitative care [1,3]. Once their condition stabilizes, inpatients typically receive daily occupational and physical therapy; outpatients visit rehabilitation clinics or receive therapist home visits several times a week for the first few months of recovery [4]. Research shows that physical recovery continues beyond six months post-stroke [3]. Unfortunately, the level of longer term care can be prematurely curtailed by patients' insurance plans, families' ability to transport patients to rehabilitative care, and patients' own motivation levels. Another significant factor limiting optimal recovery is "learned non-use" [5,6,7], viz., when stroke survivors learn to manage daily activities without using the formerly paralyzed limb even if they can.

Our work can play a critical role not only in helping stroke survivors regain physical mobility, but in helping them overcome the social, emotional and motivational barriers to doing so. Our overarching goal is to develop a domestic rehabilitative environment that is: (1) safe to use residentially, (2) engaging even for the unmotivated, (3) provides useful interface for off-site therapists and physicians, and (4) able to overcome or avoid learned non-use issues. Toward the end of the proposed period, this environment will be placed in several homes where usability and safety can be qualitatively assessed (without running the therapeutic program). This is a three-year project; after its successful completion, we intend to replicate and distribute the system to more patients' homes for complete therapeutic evaluation.

The intellectual merits of the proposed project are in: (1) the multi-disciplinary engineering contribution needed to design a novel domestic virtual robotic environment that is safe and engaging, and (2) addressing scientific questions related to useful physiological/behavioral data for off-site therapists and perceptual interactions that augment people?s movements without their conscious awareness. With these problems solved, we will be able to rehabilitate patients with motor impairments in their own homes and extend their range of motion beyond what they had previously thought possible.

The broader impacts of the proposed work are in: (1) reducing the burden on stroke survivors' families and lowering the cost of care to families and insurers, (2) extending the environment for use in domestic diagnosis, prevention, and exercise paradigms for neurological disorders, elder care, and additional disabled populations, and (3) recruiting more girls in the middle school into science and engineering by introducing the concept of ?helping people' through science and engineering. The PI is a woman with a strong track record in providing K-12 outreach.

Interacting with the physical environment and manipulating objects is an essential part of daily life. This ability is lost in upper-limb amputees as well as patients with spinal cord injury, stroke, ALS and other movement disorders. These people know what they want to do as well as how they would do it if their arms were functional. If such knowledge is decoded and sent to a prosthetic arm (or to the patient's own arm fitted with functional electric stimulators) the lost motor function could be restored. The decoding is unlikely to be perfect however the brain can adapt to an imperfect decoder using real-time feedback. Several groups including ours have recently demonstrated that at least in principle this can be achieved. However, as is often the case in science, the initial work has been done in idealized conditions and its applicability to real-world usage scenarios remains an open question. The goal of this project is to bring movement control brain-machine interfaces (BMIs) closer to helping the people who need them, and at the same time exploit the rich datasets we collect in order to advance our understanding of sensorimotor control and learning. This will be accomplished by creating hybrid BMIs which exploit information from multiple sources, combined with modern algorithms from machine learning and automatic control. RELEVANCE (See instructions): Being able to interact with the physical environment and manipulate objects is an essential part of daily life. Brain-machine interfaces are one way to restore this ability to patients who have lost it. The proposed project will bring brain-machine interfaces closer to helping patients in real-worid object manipulation tasks.

DESCRIPTION (provided by applicant): We request funds to purchase a state-of-the-art robotic arm and hand from Shadow Robot. The device is actuated by 48 air muscles whose compliance and force-length characteristics are similar to human muscles. Furthermore the kinematic structure and range of motion are similar to the human upper limb. The robot will allow us to implement and test models of sensorimotor control. Such models are traditionally based on data from simple experimental tasks, and whether they can scale to the challenging control problems which the brain solves in everyday life remains an open question. The only way to address this question is to try to control a humanoid robot in the way we think the brain controls the body, see where our current ideas fail, and find ways to improve them. In addition the robot will facilitate research in novel technologies for neuro-prosthetic control, especially control of complex hand movements which are poorly understood. The robot will also be used in research on closed-loop brain-machine interfaces.

The objective of this research is to develop algorithms and software for control of complex robotic devices. The approach is to combine online re-planning and offline learning within a novel mathematical framework, yielding richer and more adaptive behavior than what is currently possible.

Intellectual Merit

Online trajectory optimization is the method of choice for controlling slow and smooth dynamics such as chemical processes. However robot dynamics are much faster and non-smooth, presenting formidable challenges to existing methods. The proposed work will overcome these challenges, by leveraging a new mathematical framework for stochastic optimal control where the problem is reduced to a linear equation even though the underlying dynamics are nonlinear. These algorithms will use a new physics engine that relies on parallel computing to simulate robot dynamics orders-of-magnitude faster than real-time.

Broader Impact

The proposed work will change how robots and animated characters move. Currently many robotic systems are controlled in open loop, or are designed to execute one specific task well but cannot be versatile. Animation is mostly hand-drawn or based on playback of motion capture data. This work will enable both robots and animated figures to express more natural and versatile movements. Another important contribution is to neuroscience and biomechanics, where many researchers believe that the brain controls the body optimally, yet it is difficult to predict what the optimal movements are in complex tasks. The algorithms developed here will generate such predictions, and enable quantitative model-data comparisons advancing our understanding of sensorimotor brain function.

The objective of this research is to develop algorithms that can make robots and simulated characters move like humans. A range of dynamic locomotion tasks including walking, running, getting up and climbing, as well as task variations such as walking backwards, and concurrent tasks such as holding a cup of water while walking, will be studied. The approach is based on optimal control theory. Human movements will be analyzed, and the performance criteria with respect to which they are optimal will be identified. Algorithms that optimize the same performance criteria will then be developed.

Intellectual merit: Movement analysis will be based on a new mathematical framework where inference of performance criteria from observed movements becomes a convex optimization problem. Control synthesis will exploit new algorithms for real-time optimization which are able to plan long movement sequences involving multiple contact events. These algorithms rely on novel formulations of the physics of contact which are more amenable to numerical optimization, as well as a new physics simulator which exploits advances in parallel processing.

Broader impact: This research will change how robots and simulated characters move. Currently many robotic control systems with the appearance of dynamic movements are controlled in open loop, or are designed to execute one specific task. This work will enable robots to express more natural and versatile movements, as well as make robot programming more automated. The resulting controllers will also serve as models for human motor control.