Todd D. Murphey, Ph.D. - US grants

Abstract

Multiple point contact is common in many manipulation tasks. Examples
include arrays for distributed or micro manipulation, vehicles, and
traditional robotic grasping. Although all of these applications have
received a great deal of attention, no previous works have provided an
analytical approach capable of dealing with the inherent uncertainties in
the contact state--the state that represents whether a given contact is
sticking, slipping, or is out of contact. These nonsmooth transitions
between contact states have a dramatic impact on the dynamics; hence, it
is important to systematically mitigate the negative performance these
nonsmooth effects induce. Moreover, these systems are often overactuated,
so that each contact interface is independently articulated. This leads
to mechanisms that are nominally kinematically overconstrained--that is,
the kinematic relationships between all the point contacts cannot be
simultaneously satisfied. Which constraint is broken is sensitive to
details of friction and normal force modeling, so it is necessary to
estimate the current contact state and incorporate the contact state into
the motion planning and control. The goal of this research is to
produce hybrid estimators, motion planning algorithms, and control
strategies that will work in concert to guarantee performance and
stability in the face of multiple contact interfaces in an unstructured
environment. An overconstrained multiple point manipulator prototype will
be developed and planning and control methods will be applied to this
system to demonstrate manipulation that is not sensitive to the
particulars of the frictional interfaces.

Manufacturing often involves the need to reposition and reorient objects
for purposes of assembly. To accomplish this, multiple actuators are
often used, and these actuators experience stick and slip contact with the
object due to frictional interactions. The physics of the actuators and,
in particular, their interaction with the object are notoriously difficult
to model accurately. Hence, there is a strong need for manipulation
strategies that are not sensitive to these low-level details. Moreover,
many vehicles, such as the original Mars rover, have a mechanical design
that guarantees that some of the wheels must slip during operation.
However, which wheels slip is dependent on unknown environmental conditions. Hence, in this situation as well there is a need to develop motion planning strategies that are guaranteed to work even in the presence of substantial uncertainty arising from the environment. This project will contribute to these needs by developing
strategies that have guaranteed performance in the face of inherent
uncertainty. In the short term this project will contribute to macro-scale
manufacturing and vehicle control, and in the long-term will likely impact
micro-scale manufacturing.

Proposal Number: 0820004/0819929

TITLE: Abstraction-Based Motion Programs for Complex, Interconnected Systems

PIs: Magnus Egerstedt and Todd Murphey

Systematic approaches to abstraction-based motion control of complex, physical systems are still largely missing from the control-theoretic foundations of embedded system design. Examples of high-degree of freedom physical systems include humanoids, multi-leg robots, minimally-invasive surgical robotics, and cooperative systems. This work aims at understanding how high-level motion program languages can be made to form a basis for an effective software system for such complex, interconnected mechanical systems. For this, novel tools and techniques are to be developed along the following directions: 1) construction of motion description languages based on optimal control techniques; 2) development of a novel, graph-based representation of mechanical systems that allows for a compact representation of mechanical systems for simulation and analysis; 3) automatic generation of dynamically feasible motion primitives from empirical data; 4) development of an experimental testbed based on autonomous marionette puppets that can execute the developed motion programs--this testbed will also serve as a unique learning environment for students in that it requires an understanding of highly nonlinear dynamic systems, networking architectures for synchronization, hybrid systems, and optimal control.

Puppet choreography is a highly-developed language for controlling mechanically complex marionettes. It has evolved over centuries into a largely standardized form that allows puppeteers to address issues that arise as a result of the complex systems with which they are working. The project looks at how puppeteers address complex tasks in their choreographic descriptions of plays and using that understanding to solve questions of importance to computer science and engineering. These goals will be achieved by creating an automated puppet play, which will use insights about puppet choreography to implement embedded control of mechanically complex marionettes engaged in complex coordination tasks.

This work will impact a broad spectrum of activities, including integrating the choreographic structure of programming into two innovative classes, introducing puppeteers to technical computer science and engineering problems, and the introduction of puppetry as programming to children involved in a local YMCA. Students in these classes will also be statistically assessed for their ability to transfer the choreographic techniques to novel problems as well as the puppeteers? ability to apply their expertise to high-level engineering problems in order to gain insight into educational aspects of the creative process.

Hybrid mechanical systems arise in many applications, including hopping, walking, and climbing robots where contact with the ground changes; skid-steering vehicles where Coulomb friction introduces stick/slip behavior; and prosthetic devices that interact with objects. All of these applications are influenced by nonlinearity, nonsmooth transitions, and uncertainty, and these systems demand new tools in motion planning and control.

This work takes advantage of the specific structure of mechanical systems to bound the propagation of uncertainty and to develop feedback controllers that maximize robustness of execution. The work builds on state-of-the-art techniques in motion planning and estimation, including sample-based and optimization-based planning, leading to tools for uncertain hybrid mechanical systems that are analogous to control and estimation tools used for linear systems.

Example systems are used to drive algorithm development as well as to verify performance. These include 1) the Monkeybot, a robot that uses electromagnets and a single motor to locomote along a vertical wall;
2) a parkour robot that uses mechanical contact and jumping to climb narrow passages; and 3) a skid-steered vehicle that experiences discontinuous dynamics due to stick/slip friction effects. All phases of this work include participation of undergraduates and minority students. In addition to dissemination in conferences and journals, results are disseminated on a publicly viewable wiki.

Proposal #: 12-29566
PI(s): Lynch, Kevin M; Argall, Brenna; Colgate, J. Edward; Murphey, Todd D; Wu, Ying
Institution: Northwestern University
Title: MRI/Dev.: Bimanual Robotic Manipulation and Sensory Workspace

Project Proposed:
This project, developing an instrument consisting of robot arms/arms and vision-related equipment, aims to advance research in dexterous dynamic robot manipulation. The major components of the system are a two-arm manipulation system consisting of two 7-DOF WAM arms and three-finger hands with tactile and force-torque sensors; a sensory workspace consisting of high-speed vision for object tracking and color-depth cameras for lower-speed color imaging and occupancy maps; and a user command and control workstation, all integrated using software running under the Robot Operating System (ROS). The instrument enables research in various areas, such as manipulation, haptics, learning-by-demonstration, gesture recognition, rehabilitation, prosthetics, and novel sensing modalities (e.g., active electrosense).

Broader Impacts:
The area of human-robot interaction should gain much from this instrument. An active area of research, dual-arm manipulation is extremely relevant in the context of manufacturing, which constitutes an important and urgent national concern. In terms of outreach and the involvement of under-represented groups, the team?s track record is evidenced by institutional programs, like summer research opportunities and partnerships to Girl Scout troops and local science museums. This project aims to place PhD students involved in the project as interns at Barrett Technologies, thus providing opportunities to closely collaborate with the robot arm/hand designer. Such collaboration transcends the traditional vendor-buyer relationship to possibly co-design and co-publish material and software components.

The aim of this award is to develop computationally tractable methods for designing and controlling systems that experience impact. Impacts introduce discontinuities into the equations of motion, making traditional techniques in control theory difficult to employ. However, if a system can specify its configuration during impact a form of feedback control may be utilized. The work will use a combination of variational analysis for mechanical systems, shape optimization, and control theory to synthesize embedded controllers for a class of impacting systems with guarantees on stability, optimality, and computational complexity. The work will start with a simple example of a two-legged robot walking on terrain; later examples will include numerical models of high degree-of-freedom robots walking on uneven terrain and a biomechanical model of a human hand interacting with objects. Deliverables include mathematical proofs of algorithm properties, a software implementation of the feedback synthesis techniques in a variety of documented examples, articles describing the techniques, and training for students in control engineering and biomedical engineering.

Impacts are present in applications ranging from robotic locomotion and grasping to manufacturing and even human prosthetics. Computer control of these systems depends on incorporating impacts into algorithms that stabilize motion. If successful, the proposed techniques will make practical regulation of many impacting systems as straightforward as regulation of non-impacting systems. Moreover, the techniques will provide an improved understanding of the role that contact plays in human neuromuscular control. The project will leverage international collaboration through NSF Office of International Science and Engineering (OISE) Global Venture Fund (GVF) co-funding to develop long-term collaboration with researchers in Germany. Software developed as part of this work will be incorporated into an ongoing clinical study of hand rehabilitation at the Rehabilitation Institute of Chicago and will be freely available to the public. In addition to enabling important technologies, the work will be highlighted in a museum, and student will participate in outreach with an urban high school in Chicago.

The objective of this research project is to develop efficient and robust numerical methods for searching and exploring an environment using a mechanical system, while ensuring that the average temporal coverage reproduces a prescribed spatial distribution. This is achieved by developing methods for controlling mechanical systems so that they are ergodic with respect to the given spatial distribution, and combining them with geometric structure-preserving numerical integrators which have good backward error properties, and preserve geometric invariants like the symplectic structure, energy, and momentum. Furthermore, these methods preserve the nonlinear structure of the configuration manifold, such as the Lie group or homogeneous space structure. The key technical goals include: (i) the development and analysis of structured integrators to accurately predict ergodic properties of a given system; (ii) the development of simulation-based optimization of system parameters, and controls to maximize efficiency of ergodic search; (iii) the generalization of these techniques to Lie groups and homogeneous spaces, enabling ergodic search for a rich class of robotic systems; (iv) experimental validation on realistic systems.

These techniques will directly be applicable to a broad range of real-world industrial applications, including joint space exploration for robotic systems, fault detection in manufacturing, and optimal search, coverage, and information extraction for autonomous sensor networks. This industrial outreach will be facilitated by the release of public-domain software that will lower the barrier to adapting geometric numerical integration techniques in a variety of applications. Furthermore, we will engage in public outreach activities by teaming up with the Museum of Science and Industry in Chicago to develop an interactive exhibit demonstrating search algorithms for mechanical systems. These concrete applications serve to inspire high-school students (in particular, underrepresented minorities and women) to pursue STEM degrees, and this in turn will help to secure the long-term economic innovation and competitiveness of American industry.

Objective: How much a person should be allowed to interact with a controlled machine? If that machine is safety critical, and if the computer that oversees its operation is essential to its operation and safety, the answer may be that the person should not be allowed to interfere with its operation at all or very little. Moreover, whether the person is a novice or an expert matters.

Intellectual Merit: This research algorithmically resolves the tension between the need for safety and the need for performance, something a person may be much more adept at improving than a machine. Using a combination of techniques from numerical methods, systems theory, machine learning, human-machine interfaces, optimal control, and formal verification, this research will develop a computable notion of trust that allows the embedded system to assess the safety of the instruction a person is providing. The interface for interacting with a machine matters as well; designing motions for safety-critical systems using a keyboard may be unintuitive and lead to unsafe commands because of its limitations, while other interfaces may be more intuitive but threaten the stability of a system because the person does not understand the needs of the system. Hence, the person needs to develop trust with the machine over a period of time, and the last part of the research will include evaluating a person's performance by verifying the safety of the instructions the person provides. As the person becomes better at safe operation, she will be given more authority to control the machine while never putting the system in danger.

Broader Impacts: The activities will include outreach, development of public-domain software, experimental coursework including two massive online courses, and technology transfer to rehabilitation. Outreach will include exhibits at the Museum of Science and Industry and working with an inner-city high school. The algorithms to be developed will have immediate impact on projects with the Rehabilitation Institute of Chicago, including assistive devices, stroke assessment, and neuromuscular hand control. Providing a foundation for a science of trust has the potential to transform rehabilitation research.

This project develops algorithms that enable a robot to physically explore its environment using touch and to construct a language that it can use to describe that environment. The steps include exploring an environment while actively seeking information and then detecting potential elements of a language to describe what was touched. A secondary phase involves taking the set of language elements and compressing the language itself so that sensing, storage, and communication are all more efficient and more robust. The work will use a robot equipped with a robotic arm, hand, and fingertip sensors to describe objects and surfaces it encounters, all without any information about the objects provided beforehand. The importance of the work stems from the need for robots to operate in environments where touch is the only reliable sensory source. For instance, underwater applications often have limited visibility and dexterous manipulation can suffer from visual occlusion due to the hand itself. This research will enable robots to be more responsive to touch and more reliable in vision-impoverished environments.

A key technical tool used in this work is ergodic control, a computational technique that finds exploration strategies matching desired statistics. Symbol detection involves finding definitions of dynamic sensor evolution that minimize measures of variability. Language minimization depends on computing the entropy of a language, and finding the minimal language that has the same level of expressiveness. These three mathematical and algorithmic components need to be used in parallel during language creation, and they each have to respect physical limitations on the part of the robot (e.g., computational limitations and physical limitations). Software will be shared through the Robot Operating System (ROS) and TREP (physical simulation and optimal control software).

1637764 - Murphey

Robotic assistive devices help people execute and learn physical tasks. Sometimes these tasks are relatively simple, sometimes they are in a particular context, and sometimes they are highly dynamic and very task specific. This work will create algorithms that enable the delivered assistance to take into account algorithmic descriptions of the underlying task. As an example, walking is a highly structured task that simultaneously requires efficiency and stability during motion and must take into account terrain characteristics. The work takes advantage of task knowledge to either modify a person's motion or exert forces that help the person complete the task. This capability is relevant to rehabilitation and physical therapy, where one may wish to only minimally help a person in order to improve therapy outcomes. This work will therefore impact the development of software that supports people engaged in robot-assisted physical therapy, including people recovering from various forms of injury. The key to this work is that knowledge of a task is combined with knowledge of a person's capabilities to synthesize software decisions that ensure safety while also maximizing a person's agency during motion. Broader impact of this work includes technology transfer to rehabilitation, outreach through the Museum of Science and Industry in Chicago, classroom innovation, and industry collaboration.

The proposed work will create software-enabled, task-specific support for assistive biomedical devices. Dynamic tasks require that a combination of the robot and the assisted person be both effective and safe, and the proposed research will create algorithms and software that ensure efficacy and safety while leaving the user free to both move and exert effort. The latter is important in contexts like physical therapy, where effort is important to therapeutic impact. The proposed work will leverage recent results in real-time nonlinear optimal control techniques for human-in-the-loop systems. Specifically, sequential action control (SAC) will be used to both filter and assist human subject dynamic behavior, using a method called the Maxwell's Demon Algorithm. The work will additionally develop formal methodologies for establishing stability and performance guarantees for the proposed algorithms. Lastly, the proposed work will develop compact representations of the controlled assistance algorithms appropriate for computationally minimal embedded systems. All the work will be developed in the Robot Operating System (ROS), making the developed tools widely available to both researchers and companies. The algorithms will be tested on haptic devices and an exoskeleton. The broader impacts for this work will include outreach, technology transfer to rehabilitation, the development of courses in dynamics and analysis, and industrial collaboration. The PI is currently working with the Museum of Science and Industry, and as part of the proposed work the PI and supported students will participate in a National Robotics Week exhibit in the main rotunda of the museum with an estimated viewership of over ten thousand on-site visitors. The PI is involved in significant classroom innovations, and the proposed work will include development of courses in analysis and dynamics. Lastly, the project will include a collaboration with Ekso Bionics, leveraging and impacting their unparalleled expertise in exoskeleton development.

This project develops the theory and algorithms for autonomous navigation of mobile sensing platforms, such as unmanned ground, aerial, and underwater vehicles, so that the collected information is maximized while constraints on movement capabilities and energy expenditure are accommodated. This work facilitates the use of autonomous robots in environmental monitoring, search and rescue, surveillance and security, among other applications of societal importance. The algorithms are evaluated in field trials using unique gliding robotic fish. The project provides training for both graduate and undergraduate students, including those from underrepresented groups. Through showcasing at the Museum of Science and Industry in Chicago and offering of an open-source robotic fish education kit, the project promotes the interest of K-12 students and the general public in science and engineering. The project further facilitates transfer of software and hardware for robotic sensing to the market.

The goal of this project is to bridge the gap between the theory and practice in information-driven mobile sensing and to develop a principled theoretic and algorithmic framework for autonomous exploration in uncertain, specifically underwater, environments. The approach exploits the concept of ergodic exploration, where the underlying optimization problem is solved using methods from nonlinear optimal control. The research consists of: 1) Establishing a rigorous theoretical framework for ergodic exploration for guaranteeing solution well-posedness and stability, and developing synthesis methods for real-time control; 2) exploring active probing of flow conditions via auxiliary measurement of tracer agents to mitigate environmental uncertainty; 3) investigating collaborative exploration schemes that strike balance among performance, complexity, and robustness; and 4) conducting field experiments to validate the framework using a group of gliding robotic fish to monitor harmful algal blooms and localize sources of chemical spills.

This project will greatly extend a powerful new method for control of robots and vehicles, called sequential action control (SAC). One widely used approach to controlling complicated systems is to solve a real-time numerical optimization problem for the magnitude of the next few control pulses, where the all the pulses have a constant width. However even with powerful processors it can be difficult to compute these values fast enough. SAC addresses this challenge by instead computing the optimal width and relative start time of the next control pulse. For many problems of interest, this change in control strategy greatly simplifies computation, to the point that infeasible control problems become tractable. SAC allows analytical solutions to some problems, and speeds computations by up to eight orders of magnitude for others. SAC is naturally compatible with common features of modern control design, including hybrid systems that switch discretely between a collection of continuous dynamic behaviors; quantized systems where inputs, states, and outputs may take only a finite set of constant values; and systems with nonlinear dynamics. SAC can be shown to recover the globally optimal control signal in a number of analytically solvable cases. In other representative test cases, the computed SAC input provides performance that is numerically indistinguishable from the optimum. Optimal or near-optimal input signals are of no value if small disturbances cause the system to rapidly diverge from the desired behavior. Therefore practical controllers must also ensure that small disturbances to the controlled system cause only small deviations in the system response -- a property known as stability. This project seeks to rigorously derive SAC performance guarantees for a broad class of systems, as well as to show conditions under which SAC ensures stability. The Darwin humanoid robot will be used as a high-dimensional, nonlinear, hybrid testbed for this research. Control of the Darwin robot may be implemented in the open-source Robot Operating System (ROS), allowing a robust and verifiable SAC distribution for dissemination. The results of this project will enable greatly improved and verifiable control over systems such as rehabilitation robots, assistive devices, rotor vehicles, and driverless cars, using widely available and low-cost computing platforms such as mobile phones. Benefits to society from this project include enhanced safety and performance of these automated infrastructure systems. The project also includes classroom innovation, international collaboration, outreach activities through the Museum of Science and Industry in Chicago, and dissemination of open-source software.

The twofold purpose of this project is to develop sequential action control (SAC) into an actionable, near-universal method for synthesizing embedded real-time control as well as to provide foundational results on optimality, stability, and geometry. The method is computationally efficient and scales to high dimensional problems. Moreover, SAC extends naturally to Lie groups, common in applications such as robotics and automation. The project will address three fundamental questions. First, it will identify conditions under which SAC can be applied directly or iteratively to achieve optimal control. Second, it will derive conditions for stability. Third, it will adapt SAC to systems evolving on Lie groups, to achieve global performance for multibody mechanical systems. The broader impacts for this work include outreach, technology transfer to rehabilitation, the development of online courses in dynamics and analysis, and international collaboration. The PI is currently working with the Museum of Science and Industry, and as part of the project the PI, and graduate and undergraduates involved in the PI's laboratory, will participate in a National Robotics Week exhibit in the main rotunda of the museum with an estimated viewership of over ten thousand on-site visitors.

Most cyber-physical systems have operated in comparatively benign environments, often engineered to meet the needs of the system. For instance, in many settings robots operate in closed-off environments in manufacturing. However, as these systems are deployed in increasingly isolated environments (such as search and rescue efforts, automation in surgical devices, collaborative manufacturing) these robots will need to operate while reasoning about substantial uncertainty. Moreover, actions taken by the system impact that uncertainty. The proposed work will develop algorithms that enable a cyber-physical system to reason about what actions it must take to manage its uncertainty. A simple example of this is understanding that one must look behind and under things when searching for an object. The goal of this project is to automate the process of managing that uncertainty, however it arises. For instance, interacting with humans (such as physically interacting with a person while assisting her motion, or tracking a person during a search effort) involves uncertainty about what the person is going to do. Enabling a robotic system to actively test a person's intent, and then act according to her subsequent behavior, is key to the robot's ability to be an effective partner. This essential point, that action on the part of a cyber-physical system can be used to explicitly manage its understanding of the world, is the core purpose of the proposed work.

The proposed work will leverage recent results in information-based real-time nonlinear control. Specifically, ergodic control (controlling the ergodicity of a trajectory relative to some reference distribution) enables one to specify an objective using a density function to describe the desired spatial characteristics for a trajectory. In the context of Hidden Markov Models (HMMs) and Partially Observable Markov Decision Processes (POMDPs), this allows a system to actively probe multiple states, simultaneously considering process uncertainty, measurement uncertainty, and uncertainty associated with the Markov process itself, even when the HMM is changing over time because of new states or processes being introduced into the ambient environment. The apparent reason for the effectiveness of ergodic control in the context of reactive planning under uncertainty is that it always computes plans in the continuum, avoiding the combinatoric complexity associated with POMDPs. Moreover, the needs of the dynamic system and the cyber system are inter-related during physical motion, so the work will additionally develop methods that maintain stability with respect to both state and information. This inter-dependence creates a trade-off between the physical ability to act and the computational load on the cyber system, enabling a cyber-physical system to reduce the load on its cyber system through physical action. The research will develop algorithms capable of real-time information-based control in response to constantly changing data. One of the motivating examples will be aerial vehicles tracking unknown numbers of targets on the ground in a highly occluded environment such as a forest or an urban setting. Additionally, a robotic arm, already used for assistive device research, will be used to implement and assess the proposed methods.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The understanding of designing structured surfaces for advanced functionality, such as friction reduction, antifouling, and hydrophobicity, has significantly progressed over the years; however, the critical technical barrier to the application of these structured surfaces is the scalability in manufacturing capability. The biggest challenge in surface patterning is the process scalability, which needs to reconcile the significant scale difference between the individual feature size down to the nano- or micro-level and the large surface-to-be-textured up to the meter level. The project will investigate one vision for future manufacturing -- distributed robotic manufacturing -- to achieve scalable patterning of micro-structured functional surfaces using a cohort of mini-robots. The project will not only push the knowledge boundaries in the scientific understanding of distributed physical intelligence, machine-material interactions, and swarm control, but may open up a new and interdisciplinary research field at the intersection of manufacturing, robotics, control, and cyberphysical systems. Additionally, the project includes outreach at the Museum of Science and Industry in Chicago, open-source software, and curricular innovations, including online classes and training modules. The research will build an educational and outreach platform to enable education and workforce development for STEM educators, next generation workforce, and technical engineers.<br/><br/>The research objectives are to explore and answer three fundamental scientific questions that will enable the vision for future manufacturing in distributed robotic manufacturing. (1) Distributed physical intelligence: a new design framework will be established to distribute the intelligence among mechanical structures, analog circuits, and digital logics, as well as to design unconventional communication channels through both active and passive manners. (2) Unconventional machine-material interaction: new theoretical underpinnings will be established to investigate the machine-material interaction and the possible removal, deformation, and addition of material in this new paradigm, where the tools are extremely flexible and with significantly constrained power. (3) Swarm control: new fundamental knowledge will be generated from novel task decomposition and distribution paradigms to synthesis techniques capable of minimizing control effort, communication, and computation. Instead of top-down control of every individual mini-robot, novel methods will be established through which local rule specifications lead to global distributed pattern completion.<br/><br/>This Future Manufacturing award is supported by the Division of Computer and Network Systems (CNS) of the Directorate for Computer and Information Science and Engineering (CISE), and by the Division of Civil, Mechanical and Manufacturing Innovation (CMMI) of the Directorate for Engineering (ENG).<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.