Allison Okamura - US grants

The proposed work addresses development of novel methods for enhancing the ability of humans to perform complex and delicate manipulation tasks at a microscopic level. The proposed approach is based on a combination of off-line programming and on-line adaptation of task-specific human-computer manipulation idioms. The work builds upon this group's experience in the development of steady-hand robotic augmentation. Specific tasks from microsurgery and micro-assembly will be used for testing the ideas and evaluating the results obtained.

This project models and builds systems that augment human physical capabilities for performing skilled tasks. Areas of importance for human-machine physical collaboration include micro-manipulation such as microsurgery or microassembly, and remote operation (eg in outer space or under the ocean). It is also important to study such techniques in areas where great dexterity is required, such as medical palpation.
These systems are also important for training the disabled and in various educational applcations.

Human-machine collaborations of this sort, where a person and a robot are both "holding" the same knife or stick, differ from traditional interfaces in their richness of sensory inputs and the coupled computation and external physical reality. The inclusion of the human in the feedback loop makes these systems more complex than older robotics applications; human expectations and reactions must be modeled and provided for in the system. Depending on the application, vision, sound, and force-feedback may be involved; reaction times may vary; and the robot manipulator may be used to add stability, micro-scale capability, and/or safety to the system.

DESCRIPTION (provided by applicant): Cardiac surgery is traditionally performed through a median sternotomy, providing optimal access to all cardiac structures and great vessels, but generating significant disfigurement and pain for the patient. The advent of robot-assisted minimally invasive surgery (MIS) holds great promise for improving the accuracy and dexterity of a surgeon while minimizing trauma to the patient. However, clinical success with robot-assisted cardiac MIS has been marginal; improved patlent outcomes and mitigated costs have not been proven. We hypothesize that this is due in large part to the lack of haptic feedback presented to the surgeon. Without the sense of touch naturally used in surgical tasks such as fine suture manipulation, surgeon performance is jeopardized. The general objective of the proposed research is to acquire and use haptic information during robot-assisted minimally invasive surgery, with a focus on manipulation of fine sutures. It is anticipated that this approach will offer two main benefits over current systems: (1) forces will be fed back to the user in real time, directly or through sensory substitution, improving task performance, and (2) haptic information will be used to create automatic virtual fixtures that assist the surgeon, e.g., by prevention of excessive applied suture forces and maintenance of constant retraction forces. The specific aims are: (1) to understand the sensing requirements for minimally invasive surgical tasks, (2) to test different modes of haptic feedback in robot-assisted minimally invasive surgical environment, (3) to create virtual fixtures that augment and improve the execution of surgical tasks, and (4) to apply feedback of haptic information during phantom experiments, creating a direct path to clinical applications. These specific aims contribute to a Iong-term research plan for using haptic information in robot-assisted minimally invasive surgery. The proposed work represents the development of operative technology that can provide significant improvements in patient outcomes, as well as Iay the groundwork necessary to address several other exciting research issues such as beating-heart (off-pump) procedures, procedure and tissue models, and virtual environments for training.

Information Technology is making great progress in the operating room. In the practice of minimally invasive surgery, three-dimensional modeling techniques are enabling preoperative planning, new image acquisition and display techniques, and superior dexterity through teleoperated robotic systems. However, one sensory channel is presently ignored in these technological improvements: touch. In manual surgeries, haptic feedback is crucial for palpation, suture manipulation, and detection of puncture events. Sensation of forces is particularly important for invasive tool-tissue interaction tasks such as grasping, cutting, dissection, and percutaneous (GCDP) therapy. Lack of haptic information overloads the visual sensing required of the surgeon, requiring a demanding level attention. For information technology to truly enhance the practice of surgery, multiple sensory channels must be utilized.

We propose to develop instrumentation and algorithms for modeling the haptic aspect of tool-tissue interaction in four common surgical tasks, namely: grasping, cutting, dissection, and percutaneous therapy. This system will significantly enhance information display in three ways. First, data acquired in real time will be used to provide feedback to the surgeon during model-based teleoperated procedures, increasing the transparency. of the robot-assisted surgical system. Second, real-time modeling techniques will enable model-based teleoperation, removing the strict constraints imposed by time delays in traditional, direct teleoperation. Third, realistic surgical simulations will improve training, increasing surgeon competence and patient safety. The instrumentation and modeling algorithms will be used to determine parameter values for different tissue types, particularly liver, prostate, spleen, and kidney. Ex-vivo tissues and phantom hydrogel tissues will be used in these experiments. Once the instrumentation and modeling techniques are developed, we will proceed to extensive validation of the three applications of enhanced information display. Performance experiments will verify improvements in accuracy and precision for direct and model-based teleoperation, and a computer vision/force sensing system will determine the realism of force and deformation models developed for surgical simulation. This proposal addresses the ITR challenge for developing an information-enhanced display with computational, simulation, and data analysis methods for modeling common surgical GCDP tasks for which currently there is no systematic model.

Intellectual merit: The specific goal of this project is to significantly improve the information-enhanced operating room through the sensing and acquisition of models representing haptic information during tool-tissue interactions in minimally invasive surgery. This research will significantly impact: (a) tool-tissue interaction models that reflect the actual forces and deformation occurring during surgery, (b) haptic feedback to the surgeon during direct or modelbased teleoperation, thereby improving surgical outcomes, c) the development of realistic simulations for training current and future health care professionals, and d) the development of instrumented smart tools for both traditional minimally invasive and robot-assisted surgeries.

Broader Impacts: The PIs from both institutions have an excellent history of involving undergraduate students (both men and women) in research projects through Research Experience for Undergraduates (REU). Additionally, the Drexel Research Experience for Teachers (RET) site proposal recommended for funding by NSF, along with an established RET program at Johns Hopkins, will involve high-school math and science teachers from high schools to participate in summer research projects related to this proposal. These activities will enhance participation of underrepresented groups from inner-city schools and lead to broader dissemination of scientific and technological education of the students and teachers.
For graduate students, the interdisciplinary nature of this research offers new opportunities in education, broadening the interaction of mechanical engineers and medical professionals. Finally, the proposed research will lead to improvements in surgical outcomes, benefiting the patient and the society at large.

Due to the diverse areas of research required by this project, the proposed research will benefit from the collaboration of PIs from Drexel and JHU. Drexel has expertise in haptics, FEM modeling, grasping/dissection tasks, and phantom tissues while JHU has expertise in reality-based modeling for cutting/percutaneous therapies, haptics, and validation experiments for human-machine systems. The unique facilities and collaborations at both institutions, along with strong ties to medical professionals, make this work appropriate as a collaborative research project.

DESCRIPTION (provided by applicant): The long term goal of this work is to harness the effect of bevel tip needle bending to provide accurate, dexterous targeting for percutaneous therapies. New results in needle and tissue modeling, combined with robot motion modeling, will facilitate new models, hardware, control, and planning techniques for steering flexible needles inside soft tissue. Using path planning, control, and kinematics results from the field of robotics, we plan to demonstrate that an appropriately designed needle can be steered through tissue to reach a specified 3-D target. Even more compelling is that the methods we propose will allow needles to be steered to previously inaccessible locations in the body, enabling new minimally invasive procedures. The first step in needle control is to obtain an accurate model. Thus, the specific goals of this exploratory/developmental application are to under stand and model the biomechanics of needle motion. This is a high-risk activity because there has been no previous work to successfully model the 3-D path of flexible needles through soft tissue. The work is high impact because accurate 3-D needle modeling can be used to (1) improve targeting, thus enhancing the performance of many percutaneous therapies and diagnostic methods, (2) plan needle paths that steer around obstacles, and (3) develop realistic simulators for physician training and patient-specific planning. Working closely with physician collaborators, the investigators will study needle insertion scenarios that are relevant to improving the quality of health care. The specific aims are as follows: (1) Develop and validate a deterministic biomechanical model of needle insertion through un deformed tissue. The bending of the need le tip will be a function of tissue properties, needle properties and input parameters. Refine and test the model on phantom tissues. Select needle design parameters that facilitate steering. (2) Further refine the model of Aim 1 to include the effects of constraint and input uncertainties, generating a stochastic biomechanical model of needle insertion through undeformed tissue. Compute the space of possible needle tip positions for a given set of inputs and determine the probability of acquiring specific targets. (3) Further refine the models of Aims 1 and 2 to include the effects of both tissue deformation and inhomogeneous tissue properties. A particular challenge is the development of new modeling techniques that simultaneously handle needle and tissue deformation.

Using force and tactile (haptic) feedback, humans are able to accomplish complex exploration and manipulation tasks in unstructured environments, even when other sensory modes are limited. However, we cannot reach environments that are remote in space and/or scale, such as outer space, under bodies of water, and inside the human body. The proposed research activity will harness the extraordinary ability of human exploration through novel telemanipulation systems. The intellectual merit is that controls, human factors, and modeling techniques will be used to develop versatile, practical telemanipulators that provide appropriate haptic feedback to the operator and automatically generate detailed haptic models of explored environments. A set of crucial telemanipulation issues will be addressed: (1) sensor/actuator asymmetry, which is necessary for practical implementation in many applications, (2) the roles of level and degrees of freedom of haptic feedback during teleoperated haptic exploration, and (3) the design and control of virtual fixtures that assist in teleoperated exploration, while not interfering with the operator's perception of the remote environment. The broader impacts of this work are safety for humans working in hazardous environments and improved health care through robot-assisted surgery. Specific education goals are to create haptic laboratories and public displays that use the sense of touch to improve scientific understanding and interest in technology

This SGER project is a feasibility study of algorithms for manipulation and recognition using haptic information. The Manipulation-and-Perceiving-Simultaneously (MAPS) paradigm borrows key ideas from recent advances in simultaneous localization and mapping (SLAM) and visual object recognition with modifications necessary to adapt them to haptic exploration.

In MAPS, exploration is driven by the objective of locating "interest areas" that carry high information content about the object. The haptic properties of these interest points will be encoded, together with location information, in a coordinate-system independent fashion. During a learning phase, a large set of such points will be acquired and stored in a database. During recognition, newly acquired points will be compared with those in the database to generate plausible model hypotheses. The final recognition decision will also incorporate the geometric constraints implicit in the location and orientation of the surface structure.

The key contributions of this project will be:
- The development of active exploration strategies to extract, localize and map distinctive surface features;
- The development and evaluation of haptics-based object recognition algorithms using both spatial and haptic "appearance" information;
- The development of an experimental environment to implement and evaluate MAPS algorithms.

[unreadable] DESCRIPTION (provided by applicant): Minimally invasive surgical techniques have been highly successful in improving patient care, reducing risk of infection, and decreasing recovery times. This project aims to further reduce invasiveness by developing a technique to insert thin, flexible needles into the human body and steer them from outside. This approach will potentially improve a wide range of procedures, from chemotherapy and radiotherapy to biopsy collection and tumor ablation, by enhancing physicians' abilities to accurately maneuver inside the human body without additional trauma to the patient. Building on emerging methods in robotics and highly encouraging results obtained under an R21 exploratory grant, we propose to design, prototype, and evaluate a working system that will steer flexible needles through deformable tissue and around internal obstacles to precisely reach specified 3D targets. This research program will significantly advance our understanding and practice of needle therapies through integrated needle design and modeling, preoperative visualization and needle motion planning, and image-guided intraoperative needle control. The scientific and engineering advances will culminate in a set of pre-clinical trials with imaging (fluoroscopy, ultrasound, and MRI) using phantom and natural ex vivo and in vivo models. The designs, analyses, and experiments of this study will determine the merits and weaknesses of flexible needle steering, with the goals of improving current clinical applications and leading to new ultra-minimally invasive surgical procedures. The results of this project could significantly improve public health by lowering patient recovery times, infection rates, and treatment costs. By increasing the dexterity and accuracy of minimally invasive procedures, anticipated results will not only improve outcomes of existing procedures, but also enable percutaneous procedures for many conditions that currently require open surgery. [unreadable] [unreadable] [unreadable]

Proposal #: CNS 07-22943
PI(s): Okamura, Allison M.
Cowan, Noah J.; Hager, Gregory D.; Kazanzides, Peter.; Taylor, Russell H/
Institution: Johns Hopkins University
Baltimore, MD 21218-8268
Title: MRI/Dev.: Infrastructure for Integrated Sensing, Modeling, and Manipulation with Robotic and Human-Machine Systems

Project Proposed:

This interdisciplinary project, developing infrastructure for sensorimotor integration that fosters new studies and enhances existing work in the area of manipulation for robotics and human-machine systems, aims to offer a publicly available systems framework, including software, mechatronics, and integrated hardware. Enforcing a general development approach that can be easily extended to other robotic and human-machine applications, two complementary robotics platforms will built addressing two different application domains: a
. Bimanual dexterous manipulation system with integrated environment sensing and the capability for modeling rigid objects commonly found in human environments, and
. Teleoperated surgical robotic system with integrated sensors that can acquire patient-specific deformable tissue models.
Moreover, via the project's website, dissemination is planned for:
. Open-source software for real-time system control and sensor/model/manipulation/display integration;
. Design of a complementary mechatronic firewire controller board that includes A/D, D/A, encoders, amplifiers, and low-level control capabilities via FPGS, and
. Detailed descriptions of hardware integration, including WAM arms, Barrett hands, a tactile sensing suite from Pressure Profile Systems, surgical robots, cameras, ultrasound, OCT, a vision-based tracking system, visual and haptic displays, and more.

Broader Impact: Integrated robotic systems that fuse multimodal sensory information to enhance models and manipulate the environment positively impact human lives, particularly in health care, safety, and human assistance. The research also impacts related disciplines, including neuroscience, rehabilitation, and surgery. Researchers will have access to open software and designs. The platforms clearly impact education, people at many career stages, from high school students to senior faculty. The system, and detailed directions on how to produce it, will be made available. The design framework will be used in undergraduate classes in conjunction with existing educational hardware; experimental platforms will be used for course projects. Visiting students and faculty (including WISE girls) will make positive use of the integrated testbeds. There is an ongoing collaboration with Morgan State U. Involving REU and RET participants, outreach is well planned. The dissemination plan includes an active web site; the software will be made available via open-source repository. Furthermore, the website documents all hardware and respective vendors.

0651803
Cowan
The goal of this research is to develop a dexterous, small-size, cannulas type robot for Minimal Invasive Surgery (MIS) and bio-sensor delivery applications based on coaxially assembled elastic tubular structures. The movement and control of the cannulas is based on the storage and release of elastic energy via translational and rotational motions induced by the user either locally or remotely via tele-operation systems. The research will provide better understanding and characterization of how actuation can be encoded in flexible structures and how it can be controlled. It mimics various natural creatures such as snakes, octopus tentacles, elephant trunks, etc., hence it can be considered as a biologically-inspired device. Issues such as miniaturization, complex tissue-device interfacing, and tele-operation add onto the scientific challenges of the project. This activity integrates research activities with educational opportunities for undergraduate and high school students in a coherent fashion. At the same time, it seeks to develop an improved technology for minimally invasive surgery which could benefit many people through improved quality of health care.

DESCRIPTION (provided by applicant): Damage to the nervous system often results in altered movement control, leading to loss of function. The NIH estimates that movement-related neurological disorders affect millions of Americans each year. For many neurological disorders, rehabilitation therapy is a main treatment. In order to optimize rehabilitation techniques, a quantitative approach that pinpoints specific deficiencies and targets them with long-term intervention is needed. Thus, we will apply biomechanical models, robotic devices and novel control strategies for optimizing both compensation and learning approaches for improved movement control. This project focuses on poor motor coordination, which is a ubiquitous finding in people with damage to the nervous system. Incoordination leads to poor trajectory and targeting control, and is most distinctly related to cerebellar dysfunction. The fundamental mechanism of cerebellar incoordination will be investigated by comparing human performance to computational models, and then by developing robotic control strategies that either compensate for motor impairments or optimize practice-dependent learning. Both approaches are needed since short-term learning mechanisms can be inefficient or absent in individuals with cerebellar damage, making compensation the best option for some people. A robotic exoskeleton device, the KinArm, will be used to acquire behavioral data during reaching tasks performed by control and cerebellar subjects, and dynamic models of the human arm will be used to determine the source of the differences between control and cerebellar data. Specifically, the dynamic models of subjects will be used to simulate the effects of misestimation of limb dynamics and timing delays. The parameters that best explain behavior will be used to inform a rational control strategy for robot-assisted rehabilitation for ataxic populations. Adaptation and compensation methods will be designed that provide assistive and/or resistive forces to help subjects achieve normal movement patterns. A pilot study in which cerebellar patients use these methods will provide design guidelines for future rehabilitation robotics development. The long-term goal of this work is to design and produce take-home devices that are customized to either compensate for an individual's deficit or to facilitate the learning of a new motor pattern. We plan to extend this methodology to a broad range of patient populations. This project lays the foundation for novel home therapies by identifying strategies a robot could use to normalize movement control of people with cerebellar damage. PUBLIC HEALTH RELEVANCE. Movement disorders commonly occur following neurological damage, affecting the activities of daily living for millions of Americans each year. Therapies and assistance methods using rehabilitation robots are promising techniques for improving the short- and long-term health care of these patients, by lowering the cost of treatment, enabling more effective methods for practice-based rehabilitation, and providing "smart" orthoses for recovery of normal movement function in populations for whom adaptation is not possible.

While numerous studies have focused on the design of novel controllers and devices to enhance the user experience of teleoperated and virtual environments, our general understanding of what makes a haptic interface acceptable to human operators is limited. The addition of haptic feedback to a computer-mediated task, such as training to perform a surgical procedure in a virtual environment or teleoperating a robot to dispose of explosive ordnance, has been hypothesized to improve the speed, accuracy, and precision of task performance. However, the benefits of haptic feedback have been insufficient to motivate inclusion of haptics in many mission-critical systems. Recent publications in the haptics literature have shown confusing results regarding the role of haptic feedback, such as haptic teleoperators that have high user acceptance and realism ratings but do not have any effect on user performance, as well as haptic teleoperators that are disliked by users yet lead to significant improvements in performance. The PI's goal in this project is to identify the salient parameters of haptic systems leading to haptic realism and haptic utility, their trade-offs, and - hopefully - their complementary features that lead to haptic systems that are realistic and useful, and therefore acceptable to human operators in high-risk scenarios. She plans to approach this problem by examining manipulation and exploration with bilateral teleoperators, where haptic feedback consists of vibrotactile and kinesthetic feedback mediated by a tool. System parameters of interest include feedback gain and frequency content, device mechanical properties, and time delay. For each parameter, the PI will examine its effect on realism and utility, specifically: how users subjectively rate realism and their own performance, and objective measures and theoretical predictions of user perception and performance. In addition, she will survey user acceptance and measure affect, and compare these to measures of realism and utility. Based on these results, she will design and test haptic feedback systems that should maximize user acceptance. This framework will allow her to answer relevant questions such as: What is the spectrum of cost-benefit ratios in bilateral teleoperation, considering the role of device performance in defining the limits of haptic realism and utility? How does the level of risk in a task affect user acceptance of haptic interfaces? Can we design "haptic cartoons" analogous to medical illustrations, which are not completely realistic but display the most important haptic features in order to maximize task performance? And is there a haptic "uncanny valley" in which slightly unrealistic haptic feedback is particularly disturbing to operators?

Broader Impacts: Effective haptic feedback systems will improve human health and well being through applications such as surgery and explosive ordnance disposal. This project will provide the field of computer-mediated haptics with a theoretical and experimental framework to describe the effects of system design choices on user acceptance of tool-based haptic teleoperators. The framework will likely apply to other forms of haptic display (such as sensory substitution and spatially distributed tactile feedback) and in other haptic feedback scenarios (e.g., medical training and the transfer of learning in simulation to real-world tasks). Project outcomes will be disseminated through software and data made publicly available on the PI's laboratory website, a conference workshop, and a new course on haptic design. Outreach programs, public lab tours, and mentoring of female and minority graduate students, undergraduates, and high school students will broaden participation of underrepresented groups in engineering.

This project addresses a large space of manipulation problems that are repetitive, injury-causing, or dangerous for humans to perform, yet are currently impossible to reliably achieve with purely autonomous robots. These problems generally require dexterity, complex perception, and complex physical interaction. Yet, many such problems can be reliably addressed with human/robot collaborative (HRC) systems, where one or more humans provide needed perception and adaptability, working with one or more robot systems that provide speed, precision, accuracy, and dexterity at an appropriate scale, combining these complementary capabilities.

The project focuses on multilateral manipulation, which arises when a human controls one or more robot manipulators in partnership with one or more additional controllers (humans or autonomous agents). Complex operations in surgery and manufacturing can benefit from the extra degrees of freedom provided by more than two hands, and training often depends on hands-on interaction between expert and apprentice. Example applications include surgical operations, which typically involve several physicians and assistants, and other medical tasks such as turning a patient in bed and wrapping a cast to constrain a hand. Multilateral manipulation also applies in manufacturing, for example for threading wires or cables, aligning gaskets to obtain a tight seal, and in many household situations, such as folding tablecloths, wrapping packages, and zipping overfilled suitcases so they will fit inside diabolically-designed overhead airline compartments. Multilateral manipulation often arises with deformable materials or multi-jointed objects with more than six degrees of freedom (DOF). The extra DOFs in materials introduce challenges such as computational complexity, but they also can accommodate minor inconsistencies through redundancy and provide system damping. This project advances the fundamental science of multilateral manipulation guided by specific applications from surgery and manufacturing.

Broader Impacts: Multilateral manipulation systems have the potential to improve healthcare, improve American competitiveness and product quality in manufacturing, and open the door to new service robot applications in the home. The project will be guided by an Advisory Board of experts from industry and medical practice. Project results will be disseminated through yearly conference workshops, open-source software tools integrated into common robotics software environments such as Robot Operating System (ROS), and the investigators' research and course webpages, to encourage integration of our approach into research projects and courses at many institutions. Outreach programs, public lab tours, and mentoring of minority students will broaden participation of underrepresented groups in engineering. These activities will encourage participation in STEM activities and provide student and postdoctoral researchers with mentoring experience.

The Cyberlearning and Future Learning Technologies Program funds efforts that will help envision the next generation of learning technologies and advance what we know about how people learn in technology-rich environments. Cyberlearning Exploration (EXP) Projects explore the viability of new kinds of learning technologies by designing and building new kinds of learning technologies and studying their possibilities for fostering learning and challenges to using them effectively. This project's technology innovation is a haptic (force-feedback) toolkit to be used in learning science and engineering content that involves forces. The toolkit the team is developing will allow learners (K-12 and beyond) to manipulate simulations and models through touch-sensitive devices and to directly feel resulting forces. Learning and using the science of forces is quite complex, as the ways forces actually combine and have effects are not always consistent with what people experience. Understanding forces, however, is essential for pursuits as varied as engineering, rehabilitative medicine, and construction and for understanding concepts and phenomena in biology, chemistry, physics, and engineering fields. The kind of direct engagement with forces at the micro-level that the proposed toolkit will allow has potential to foster better scientific understanding across this whole set of fields. Research in the context of this technological innovation focuses on the extent to which and the conditions under which such embodied experience with phenomena fosters deeper understanding.

The role of embodied experiences in learning is both an important issue in learning how to better foster learning and a newly-emerging possibility for learning technologies. In this project, the PIs examine the potential for haptic technology to expand and transform student learning. The haptic interface being developed is a mechatronic device programmed with force-displacement relationships that a user experiences through the sense of touch. The project aims to demonstrate the ways the programming and use of haptic devices as a part of lab experiences can impact learning. The PI is a mechanical engineer who is expert in materials and devices. The co-PI is a learning scientist engaged closely with the maker movement and with helping learners learn from simulation and modeling experiences. Through an iterative, design-based research approach, new haptic devices and accompanying visual programming software are being created to enable interactive hands-on virtual laboratories for biology, chemistry, and physics, and experiments are being carried out to understand both how to use such devices well to foster learning and the affordances of haptics in learning force-related concepts and its added value in the context of learning through modeling and simulation.

DESCRIPTION (provided by applicant): An estimated 33,190 new cases of primary liver cancer and an even larger number of new cases of metastatic cancer to the liver will be diagnosed in the United States in 2014. Eradicating those tumors improves patient survival. Percutaneous radiofrequency ablation (RFA), which kills tumors via thermal damage, is a minimally invasive treatment appropriate for the many patients who are not eligible for surgical resection, particularly those with certain tumor type, location, or size; poor liver function; or oher comorbidities. However, technical limitations have precluded broader adoption of RFA as a means of eliminating liver tumors. Tumors in some portions of the liver cannot be accessed because the straight paths currently taken by RFA electrodes would traverse vasculature, lung, or other sensitive structures. Large tumors require multiple electrode insertions; the increased bleeding risk with each puncture through the liver capsule can be too great in patients with advanced liver disease or severe comorbidities. The long-term goal of this work is to significantly improve access, accuracy, and precision during percutaneous liver RFA procedures using robot-assisted needle steering. The following specific aims will enable the development of this technology and demonstrate feasibility, as a path to clinical adoption: Specific Aim 1: Optimize ultrasound-based localization of highly curved needles and closed-loop control of robotic needle steering. Specific Aim 2: Develop an interactive user control interface and enable teleoperated needle steering intervention. Specific Aim 3: Design a clinical robot-assisted needle steering system for percutaneous liver RFA, validated in an ex vivo liver model. Specific Aim 4: Validate the ultrasound-guided needle steering system in vivo in porcine liver. Needle steering enables the insertion of highly flexible needles or guidewires along controlled, curved, 3D paths through tissue. Thus, the needle can be actively steered to acquire tumors not accessible by a straight- line path, and unexpected deviations from the desired path can be corrected as the needle advances toward the tumor. A functional needle steering system appropriate for in vivo experimentation must be compatible with widely available medical imaging (here, ultrasound imaging), have the ability to deliver treatment (here, RFA through the use of a steerable needle electrode), and an appropriate user interface including visualization and human-in-the-loop teleoperation. This technology could extend the survival advantages of RFA to patients who may otherwise not have good treatment options, and in the long term, be used for accurate placement of diagnostic devices and treatment of diseases in other organs.

In contrast to legged robots inspired by locomotion in animals, this project explores robotic locomotion inspired by plant growth. Specifically, the project creates the foundation for classes of robotic systems that grow in a manner similar to vines. Within its accessible region, a vine robot provides not only sensing, but also a physical conduit -- such as a water hose that grows to a fire, or an oxygen tube that grows to a trapped disaster victim. The project will demonstrate vine-like robots able to configure or weave themselves into three-dimensional objects and structures such as ladders, antennae for communication, and shelters. These novel co-robots aim to improve human safety, health, and well-being at a lower cost than conventional robots achieving similar outcomes. Because of their low cost, vine robots offer exceptional educational opportunities; the project will include creation and testing of inexpensive educational modules for K-12 students.

This work broadens the concept of bio-inspired robots from animals to plants, the concept of locomotion from point-to-point movement to growth. In contrast to traditional terrestrial moving robots that tend to be based on the animal modality of repeated intermittent contacts with a surface, the vine modality begins with a root, harboring power and logic, and extends using growth, increasing permanent contacts throughout the process. This project will demonstrate a soft robot capable of growing over 100 times in length, withstanding being stepped on, extending through gaps a quarter of its height, climbing stairs and vertical walls, and navigating over rough, slippery, sticky and aquatic terrain. The design adopts a bio-inspired strategy of moving material through the core to the tip, allowing the established part of the robotic vine to remain stationary with respect to the environment. A thin-walled tube fills with air as it grows, allowing the vine robot to be initially stored in a small volume at its base, and to extend very large distances when controllably deployed. Mechanical modeling and new design tools will enable the development of task-specific vine robots for search and rescue, reconfigurable communication antennas, and construction. The paradigm of achieving movement and construction through growth will produce new technologies for integrated actuation, sensing, planning, and control; novel principles and software tools for robot design; and humanitarian applications that push the boundaries of collaborative robotics.

This National Robotics Initiative (NRI) project will promote the progress of science and beneficially impact human health and quality of life by developing wearable soft robotic devices with distributed tactile stimulation that enable new forms of communication. Human-robot interactions will be commonplace in the near future. In applications such as self-driving cars and physically assistive devices, interaction will require effective and intuitive bidirectional communication. Transferring information through vision and sound can be slow and inappropriate in many circumstances. This project focuses on haptic (touch-based) robotics to enable communication in a salient but private manner that alleviates demands on other sensory channels. This project serves the national interest by advancing knowledge in the fields of human perception, psychology and neuroscience, while developing novel, convergent technology that integrates concepts across the fields of robotics, haptics, and control engineering. Project results will be disseminated through tactile haptic devices for education and publicly available software and data. The project aims to broaden participation of underrepresented groups in engineering through outreach programs, public lab tours, and the mentoring of female and minority graduate students, undergraduates, and high school students.

Wearable haptic systems have the potential to enable private, salient communication between humans and intelligent systems through an underutilized sensory channel (somatosensation). In this research, information will be transmitted through the haptic channel via wearable, ubiquitous, soft robotic devices that provide both passive and active touch interactions with the human user. This research is comprised of four main objectives. First, a characterization of human perception of the forearm will set the requirements for the frequency, amplitude, directions, spacing, and temporal actuation patterns for a two-dimensional array of haptic stimulators that are able to convey a range of haptic cues. Second, the project will develop a wearable, soft, haptic device able to stimulate the skin of one forearm, while also providing mechanical stimuli that are intended to be explored by the fingertips of the other hand. Third, the project will develop rendering algorithms for the haptic device that take into consideration human perceptual abilities for passive stimulation of the arm and active exploration by the fingertips. Fourth, the project team will create application scenarios to evaluate and refine the system. Wearable haptic systems have potential to improve human health and well-being through a variety of applications including: physical cueing for rehabilitation/movement therapy; explosive ordnance defusing; feedback from assistive devices including mobile robots in the home; tactile communication to enable design and e-commerce; immersion in virtual worlds for education; and the facilitation of remote interaction between people.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

In the real world, people rely heavily on haptic (force and tactile) feedback to manipulate and explore objects. However, in virtual and augmented reality environments, haptic sensations must be artificially generated. Advances in both sensing and feedback are needed to create more realistic virtual scenarios for human interaction, education, and training. This research will implement wearable fingertip haptic feedback systems that enhance human perception and task performance in virtual and augmented reality, and will provide the field of computer-mediated haptics with important new design and rendering insights for a promising form of wearable haptic device. Successful development of fingertip haptic devices and improved tracking methods will lower the cost of integrating haptic technology into virtual reality, and yield more realistic and compelling virtual experiences. Effective haptic feedback systems for virtual reality utilizing the developed technologies will have broad impact by improving human health and well-being through a myriad of applications such as training for critical tasks like surgery and defusing of explosive ordnance, tactile communication to enable design and e-commerce, and immersion in virtual worlds for enhanced education and training. Project outcomes will be disseminated through demonstrations of educational applications, expansion of an online course on haptics, and publicly available software and data. Outreach programs, public lab tours, and mentoring of female and minority graduate students, undergraduates, and high school students will broaden participation of underrepresented groups in engineering.

This project has three main technical components. First, instrumented objects will be developed that measure force interactions and integrate that data with external measurements of finger pose. Second, new wearable haptic devices will be developed that use a combination of local kinesthetic feedback and skin deformation to provide realistic haptic feedback with minimal encumbrance. Modular haptic device designs will facilitate rapid testing of various design choices. Then the basic design will be optimized in terms of control strategy, degrees of freedom, skin contact conditions, and finger grounding to best match the real-world grasping and manipulation data collected earlier. Third, predictive tracking algorithms will be developed for grasping based on characteristic behaviors observed in real-world grasping and manipulation data. All of the haptic devices and tracking methods will be evaluated both in virtual and in augmented reality environments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project will contribute to the development of new soft robot manipulators that can assist people with limited physical abilities in their activities of daily living in the home. Soft robots are made of flexible and stretchable materials configured to increase human safety, lower cost, and mechanically adapt to their environments, which are significant advantages compared to traditional, rigid robots. However, today's soft robots are far from capable of performing household manipulation tasks; they have limited dexterity and cannot handle the weight of many objects encountered in daily life. This award supports research to develop novel mechanisms and design tools in order to create soft robots that seamlessly integrate into homes and provide needed assistance. This project aims to improve human health and quality of life by (1) assisting people with limited physical abilities in their homes, (2) lowering the cost of robot manipulators, toward making robots accessible to a wide spectrum of users, and (3) promoting human safety through controllable mechanical compliance and robustness. These features are important to improve equity in access to technology, especially in scenarios where physical distancing between humans is required. This interdisciplinary project integrates mechanical engineering and computer science, and the project will educate a diverse group of students in classes and research in order to broaden participation in STEM.

This project will develop a modular set of new soft robot components that capitalize on previously developed steerable, tip-extending inflated beams: continuum links, variable stiffness control, and variable discrete joints. Each component contributes a valuable capability toward 3D manipulation. Because the process of specifying and combining these components in order to create a robot to achieve a particular task is too difficult for human designers to effectively navigate, the research team will create simulation and planning software. This software will enable a computational design process considering inverse kinematics, inverse dynamics, and trajectory optimization to develop both the overall design and the real-time control strategy. In the simulation and planning environment, users who are not robotics experts will be able to specify the critical tasks, test the ability of the robot to achieve the specified tasks as well as new tasks, and influence the design to be compatible with their needs. The effectiveness of this approach will be demonstrated through user studies and the implementation of a variety of inflated-beam soft 3D robot manipulators that perform different household tasks.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.