Yoky Matsuoka - US grants

We are poised at the threshold of an information rich world with devices and services able to deliver that information to nearly anyone, at any place, and at any time. Humans have evolved social mechanisms for smoothly and flexibly managing interpersonal communications; however, current computational and communications devices are, almost without exception, utterly unaware of the social and attentional state of the user. They know little or nothing of the personal, social, and task situations in which they are used, and they do little or nothing to account for, and minimize, the human costs they induce. In this project, the PI and his team will explore situationally appropriate interfaces that retrieve, generate, and deliver information in a manner that is sensitive to the situation of the user. These interfaces will allow for communication and information systems that maneuver, rather than blunder, through the social world. To accomplish this ambitious goal, the team will pursue a three-part research plan. First, they will use behavioral theory and research to model social mechanisms for managing interpersonal communications. The comparatively unexploited research we will draw on examines the affordances of situations and consistent patterns of human nonverbal social behavior within situations. Second, they will extract key situational and user behavior data from these models via input from new sensing technologies, using noninvasive (e.g., vision-based) sensing technology to provide information about situations and users. Third, leveraging knowledge from sensory, perceptual, and cognitive psychology, as well as from the fields of visual and interaction design, the team will create displays and interaction designs that are far more situationally appropriate than today's interfaces. To address the substantial challenges that this breadth of work presents, the PI has assembled a strong multidisciplinary team that brings expertise from computer science, social, sensory, perceptual, and cognitive psychology, and the field of design.

DESCRIPTION (provided by applicant) In this proposed research, we intend to develop two new techniques which will be used to identify a highly effective robotic rehabilitation strategy. Animal models will be used to address issues that cannot be addressed using human patients. Currently, several robotic stroke rehabilitation techniques are being evaluated to determine their effect on human patients' short and long term recovery performance. Robots, for example, are being attached to patients' limbs and applying force to move them as physical therapists routinely do. Such robotic techniques, however, are simple extensions of what physical therapists are already doing, and the only outcome measurements available are patients' behavioral changes. Robots are currently being used on a limited basis in stroke rehabilitation research because it is not ethical to test a variety of robot force fields or techniques on humans if these fields or techniques have not been proven to have a positive effect on them. Therefore, we believe that evaluating robotic rehabilitation techniques on animal models is crucial. To our knowledge, animal models have never been used to evaluate robotic rehabilitation of stroke. To use animal models, we must develop two new techniques that have not yet been explored. First, we will develop a technique to produce a precise lesion in an animal that simulates a stroke without risking the animal's survival rate. To do this, we will use a non-invasive photochemical technique. Second, we will design, construct, and test a new robot controller technology for animals. We will rehabilitate animals using this new robotic controller which will later be applicable to human rehabilitation techniques. We will combine these techniques to establish the superiority of robot-assisted intervention over non-assisted rehabilitation, explore the optimal training schedules, and identify gene products that are selectively modulated following robotic rehabilitation. The results generated in this project will be used as preliminary results to apply for an R01 grant in which effective robotic force assistance will be investigated to identify the optimal therapeutic solution for robotic rehabilitation. We have no doubt that the experimental results we produce with these techniques will significantly affect the field of rehabilitation.

Proposal Title: PECASE: Virtual Robotic Environments for Rehabilitation and
Human Augmentation
Institution: Carnegie-Mellon University

To develop truly integrated human-robot interactive environments to understand, assist, rehabilitate, and enhance the human neuromuscular systems, a virtual robotic environment that explores human's full potential needs and abilities will be developed. As the first step, an inherently safe stationary wearable arm system will be designed and constructed with a neuro-muscular system identification technique to extract the human adaptation parameters in real time. The intellectual merit of the proposed activity is that, using this virtual robotic environment, fundamental scientific questions relating to human perceptual and neural interactions between the virtual and actual movements will be investigated. With these intellectual contributions, this environment will immediately lead to rehabilitating patients with motor impairments, even for those who are not strong enough to execute task-level movements on their own, by steering them beyond what they had previously thought they were capable of doing. The educational goals are to create interdisciplinary educational environment, to increase the number of women in engineering and computer science, and to increase the interaction between motor-impaired students and engineering students. The proposed research and educational approach will have a significant and broader impact not only in the scientific and educational communities but also the lives of disabled people who are not currently able to independently execute everyday tasks.

This project was originally funded as a CAREER award, and was converted to a Presidential Early Career Award for Engineers and Scientists (PECASE) award in September 2004.

This project, investigating human grasping and manipulation strategies to gain insight into synergies employed by human subjects in grasping and manipulation tasks, aims to explore how best to use these findings to create better control algorithms for robot hands. Of particular importance are techniques to make autonomous robot behavior robust to uncertainties and to make algorithms with a human in the loop, such as teleoperation and control of prosthetic devices, more intuitive and effective for fine manipulation tasks. Although a high degree of freedom (DOF) device may be required to manipulate a wide variety of objects and perform a wide variety of tasks, in any given task situation, only a small number of independently controlled DOF may be necessary. To make significant progress towards dexterous grasping and manipulation, we must:
Make the best possible use of available sensing technology (specially for force sensing), and
Understand how to analyze, plan, and control hand motion in a reduced degree of freedom space (take advantage of task-based coordination rules and synergies that may make real time grasp optimization and planning tractable).
The infrastructure should contribute in answering the following questions:
What is an optimal grasp, and how does it depend on the kinematic and dynamic properties of the device doing the grasping?
What is the relationship between critical signals, muscle activation levels, hand shape, and force production during task performance?
Does analysis of data collected along the pipeline from cortical signals to force production allow easy organization into grasp primitives or result in "control handles" that a human operator of a robot hand would find intuitive?
How can human demonstrations of grasping and manipulation tasks be employed as the wonderful resource they seem to be, i.e., how can individual examples be converted into control algorithms that will function on a robot and be robust to variations and uncertainties?

Broader Impact: The results will improve the understanding of human hand motion and force production, the use of force information in teleoperation and control of prosthetic devices, and the ability to coach robot behavior through task demonstration. Teams of undergraduate students will use the facility and data collected will be made available on the web.

DESCRIPTION (provided by applicant): We propose to complete the construction and validation of the anatomical robotic finger within the R21 funding structure as a way to continue this research using the R01 funding structure to investigate the human hand neural strategies and to help design FES and prosthetic hand controllers. We will focus on completing and validating the robotic index finger. The specific aims of this proposal are to: 1) Incorporate the robotic skin that will be used as the feedback device during object exploration for the index finger. This will complete the mechanical construction of the robotic finger. 2) Identify the system component (the relationship between external force perturbation and joint displacement before cortical feedback) for both the anatomical robotic index finger and the human index finger. We will compare these components and iterate on the robot's mechanical design. 3) Identify the active system component (the relationship between muscle tension (motor current) and joint displacement) for the robotic index finger given no external perturbation. Using this active system identification database, we will determine combinations of muscle tensions that would achieve a specified movement trajectory. Furthermore, we will select several cost functions that could be used to achieve good engineering solutions to control this robotic finger. When we accomplish the above, we will have the first mechanical model of the index finger that can be controlled using "engineering" solutions. Once this system is constructed, we will be able to compare between the biological and the engineering solutions used to achieve the same finger movement. This comparison will allow us to infer the muscle synergy and optimization criteria used in the nervous system. In addition, we will be positioned to investigate the neural control of the movement with the robotic index finger during object manipulation with cutaneous feedback. For example, we will be able to investigate the neural control strategies used when pushing objects of different weights and sizes from one place to another. We will also explore how we take advantage of the finger's passive mechanisms to manipulate objects. Finally, we will be able to expand our system to include the thumb to analyze pinching, other fingers to study the relationship between different finger interactions during manipulation, and the palm to understand the role of the palm during manipulation. We believe that this hand will not only be a tool to investigate the neural control of movement, but it will also be used as a foundation to construct a future prosthetic device that will seamlessly communicate with the nervous system.

[unreadable] DESCRIPTION (provided by applicant): Due to advances in modern medicine, the elderly population is growing worldwide, and, along with this growth, there is a growing need for physical rehabilitation after strokes, which have an average onset of over 65 years of age. Given the magnitude of this problem and its societal ramifications, the time is ripe to explore the extent to which robotic devices and virtual environments can be used as a means of rehabilitation to improve the quality of life for both the elderly and the physically disabled. According to recent studies in robotics, robot-assisted stroke rehabilitation enhances arm movement recovery. Moreover, robot-assisted rehabilitation improves patients' mobility and strength to the point where it is equal to, or greater than, that which is achieved by human-assisted therapy. However, none of the currently available systems addresses patients' perceptual or cognitive deficits. Furthermore, these systems neglect to address the fact that many patients do not reach their full mobility potential using these systems. To remedy these problems, a virtual robotic environment that explores the full potential needs and abilities of patients must be developed. We will coin this strategy, "rehabilitation by distortion." To develop an environment to rehabilitate by distortion, however, there are two fundamental issues to address. First, it is necessary to quantify the perceptual gap that can be created between virtual and actual movements that is not perceptible to patients. To do so, we will first quantify the lowest sensory resolution, also known as just noticeable difference (JND), of force and position. As we explain below, the JND will act as the lowest bound of the perceptual gap. In addition, we will quantify the size of the perceptual gap with the existence of visual feedback distortion. Second, having identified a perceptual gap that is not noticeable, we must prove that mobility and strength of stroke patients can be extended by undetected distortion. For this proposed work, we will isolate working with one finger. Fingers are one of the parts of the body that are most commonly affected by strokes; thus testing the basic concepts about the perceptual gap between virtual and actual movements using fingers is appropriate. After we prove that rehabilitation by distortion is therapeutic for a finger, we can expand our work to other limbs. In addition, to show that this strategy is effective for patients who already received traditional therapy, we will work with patients who have already completed their traditional therapy and are at least one year after the onset of strokes. If we prove that we can allow patients to move beyond what they thought was possible after traditional therapeutic techniques, the results will be groundbreaking and will lead to an R01 grant to develop a new robotic virtual therapeutic strategy, "rehabilitation by distortion," that extends the force production and range of motion for motor impaired patients recovering from stroke. [unreadable] [unreadable]

This project aims to reverse-engineer the human brain's ability to control the hand.
The project begins by combining a robotic hand previously developed by the PI with a new type of sensitive skin, with a hundred biomimetic tactile sensors.

The main goal of this project is to understand how it is possible to achieve dextrous, approximately optimal control of a hand, performing familiar but challenging tasks in manipulating objects. New, more advanced learning-based control algorithms will be developed and tested on the four empirical testbeds of the project: (1) robotic manipulation by the biomimetic hand; (2) data from recording of human hands performing the same tasks; (3) computer simulations of physical hands; and (4) computer control of cadaver hands via their tendons. The project will use the same algorithms both as models of human motor control and to go beyond the present state of the art in robotic manipulation; this unified approach to biology and engineering is an essential part of the transformative goals of the COPN topic. Dextrous robotic hands have a wide variety of possible applications in industry, space and national security. Improved understanding of how humans can learn to perform better with their hands will also have broader benefits, particularly for the disabled. The team proposes a vigorous plan for education and outreach, capitalizing on the human interest aspects of the demonstrations they will be developing.

The field of neural engineering aims to develop technologies that discover, rehabilitate or alter brain function while at the same time deriving inspiration from neurobiology to build artificial systems with adaptive capabilities similar to biological systems. The ultimate goal is to enhance quality of life both for disabled and healthy individuals. Neural engineering solutions will reduce the economic burden on families and communities and bestow a greater degree of freedom and confidence.

This is an opportune time to unite the efforts of academia and industry in the Pacific Northwest region and create a Center for Neural Engineering. As a first step, we propose to run a workshop in which local researchers can learn about each other and the plan for the center, listen to international experts in the field, and exchange ideas through brainstorming sessions. We have invited keynote speakers who are recognized experts and from outside the Pacific Northwest and others who may serve as future advisory board members for the center. Our aims include using the workshop as a launching pad for the center and leveraging the visibility gained to initiate the resource gathering process both regionally and nationally.

With the growth of the elderly population, finding a way to enhance quality of life and safety with minimum human assistance will become critical in the next few decades. Japanese robotics researchers have been focusing on research and applications in these areas in light of the aging Japanese population. U.S. robotics researchers have also been working on the same areas but from different viewpoints. Even though both Japan and the U.S. are conducting research on the same areas, there has been virtually no collaboration between U.S. and Japanese researchers in these research areas due to the lack of appropriate funding structures and mechanisms.

The goal of this workshop is to initiate and facilitate collaborations between Japanese and U.S. researchers in the areas of robotics for safety, security and society. The proposed workshop will bring 16 Japanese researchers and 16 U.S. researchers together to discuss their research, and to explore mechanisms for joint collaborations to advance research in and improve Safety, Security, and Society. As the collaborative research evolves, enhanced research results will be published as collaborative work, applications will affect society, and ideas will be generated for future collaborative structures.

0930908
Rao


Brain-machine interfaces (BMIs) are devices that allow a subject to control objects directly using brain signals. Such devices offer the potential to significantly improve the quality of life of locked-in, paralyzed, or disabled individuals by allowing them to communicate via virtual keyboards and control prosthetic robotic devices. The two dominant paradigms for brain-machine interfacing today rely on non-invasive recording from the scalp (EEG) and invasive techniques based on intracortical implants. EEG signals are extremely noisy, thereby limiting the bandwidth of control signals that can be reliably extracted. Intracortical implants on the other hand yield stronger signals but pose serious health risks.

In this proposal, the PI describes a research program for investigating BMIs based on electrocorticography (ECoG), a relatively new technique that involves recording signals subdurally from the brain surface. These signals have much higher signal-to-noise ratio than EEG signal while at the same time, pose lesser risks than techniques that penetrate the brain surface. The proposed research will address the following key issues:

(1) Exploiting high frequency ECoG signals for BMI: Recent work has shown the existence of broad-spectral ECoG changes at high frequencies during movement and imagery. The PI and his team will explore the application of such ECoG modulation for multi-dimensional control in BMIs.
(2) Neural plasticity of local cortical circuits during BMI: The PI's team will investigate the dynamic range of the spectral changes in ECoG and analyze the adaptations that occur due to brain plasticity during BMI control. This will help pave the way for controlling 3 or more degrees of freedom in a BMI from a single control electrode.

(3) Abstraction of control signals: After extended periods of BMI use, many patients report no longer imagining moving a control limb but rather concentrating on the desired result of the BMI task itself. The PI and his team will explore the creation of new cortical communication pathways underlying such abstraction and leverage these new control signals in expanding the bandwidth of the BMI.
(4) Applications of new control signals to novel BMI paradigms: The BMI techniques will be tested using virtual devices such as cursor-driven menu systems for communication as well as more complex robotic systems such as a prosthetic robotic hand and a humanoid robot.
The educational component of the project involves curriculum development, interdisciplinary training for graduate and undergraduate students, and outreach to K-12 students.

Intellectual Merit: The proposed research represents one of the first efforts to exploit ECoG and the brain's plasticity to build BMIs that can control devices with large degrees of freedom. The study of abstraction of control signals and its application to robotic BMIs is also novel.
Broader Impact: If successful, this research will lead to new ECoG-based BMI systems that will surpass the abilities of current BMIs by relying on the brain's ability to adapt to novel control scenarios and leveraging the large-scale population-level electrical activity measured by ECoG. The project will enable the training of graduate students in a multidisciplinary environment. Promising undergraduates, including students from underrepresented groups, will gain valuable research experience in preparation for industrial and academic careers. A K-12 outreach effort will enable students from local area schools to visit the laboratories of the PIs and gain hands-on experience in the emerging field of brain-machine interfaces.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

0930927
Matsuoka

About 700,000 Americans suffer from new or recurrent strokes each year, with some 500,000 being fortunate enough to survive [1]. Over half of all strokes occur in areas of the brain that control movement, leaving many victims with impaired motor functions [2] that are most often localized in one side of the body and in one specific area, such as the arm or hand.

With careful physical rehabilitation, damaged motor functions can recover partial or full mobility as the nervous system rewires its neural circuits to represent lost functions at new neural locations. The level of recovery depends on the amount and quality of post-stroke rehabilitative care [1,3]. Once their condition stabilizes, inpatients typically receive daily occupational and physical therapy; outpatients visit rehabilitation clinics or receive therapist home visits several times a week for the first few months of recovery [4]. Research shows that physical recovery continues beyond six months post-stroke [3]. Unfortunately, the level of longer term care can be prematurely curtailed by patients' insurance plans, families' ability to transport patients to rehabilitative care, and patients' own motivation levels. Another significant factor limiting optimal recovery is "learned non-use" [5,6,7], viz., when stroke survivors learn to manage daily activities without using the formerly paralyzed limb even if they can.

Our work can play a critical role not only in helping stroke survivors regain physical mobility, but in helping them overcome the social, emotional and motivational barriers to doing so. Our overarching goal is to develop a domestic rehabilitative environment that is: (1) safe to use residentially, (2) engaging even for the unmotivated, (3) provides useful interface for off-site therapists and physicians, and (4) able to overcome or avoid learned non-use issues. Toward the end of the proposed period, this environment will be placed in several homes where usability and safety can be qualitatively assessed (without running the therapeutic program). This is a three-year project; after its successful completion, we intend to replicate and distribute the system to more patients' homes for complete therapeutic evaluation.

The intellectual merits of the proposed project are in: (1) the multi-disciplinary engineering contribution needed to design a novel domestic virtual robotic environment that is safe and engaging, and (2) addressing scientific questions related to useful physiological/behavioral data for off-site therapists and perceptual interactions that augment people?s movements without their conscious awareness. With these problems solved, we will be able to rehabilitate patients with motor impairments in their own homes and extend their range of motion beyond what they had previously thought possible.

The broader impacts of the proposed work are in: (1) reducing the burden on stroke survivors' families and lowering the cost of care to families and insurers, (2) extending the environment for use in domestic diagnosis, prevention, and exercise paradigms for neurological disorders, elder care, and additional disabled populations, and (3) recruiting more girls in the middle school into science and engineering by introducing the concept of ?helping people' through science and engineering. The PI is a woman with a strong track record in providing K-12 outreach.

Over the last decade, the field of neural engineering has demonstrated to the world that a computer cursor, a wheelchair, or a simple prosthetic limb can be controlled using direct brain-machine and brain-computer neural signals. However, technologies that allow such accomplishments do not yet enable versatile and highly complex interactions with sophisticated environments. Today's intelligent systems and robots can neither sense nor move like biological systems, and devices implanted in or interfaced with neural systems cannot process neural data robustly, safely, and in a functionally meaningful way. Doing so requires a critical missing ingredient: a novel, neural-inspired approach based on a deep understanding of how biological systems acquire and process information. This is the focus of this proposal.

The NSF ERC for Sensorimotor Neural Engineering (ERC/SNE or "Center") will become a global hub for delivering neural-inspired sensorimotor devices. Using devices that mine the rich data in neural signals available from implantable, wearable, and interactive interfaces, the ERC/SNE will build end-to-end integrated systems. Examples include: implantable neurochips that can activate paralyzed limbs by electrically stimulating muscles or nerve roots; stationary robots that extract neural signals from a user's touch to provide home-based, post-stroke therapy; neural-controlled adaptive prosthetic limbs that provide sophisticated sensory feedback, and wearable caps that control external exploration devices. Unlike traditional approaches that stress accommodation to the needs of people with neurological disabilities, the ERC/SNE will focus on proactive technologies that provide seamless and adaptive person-machine interaction. It will accomplish this mission with three core engineering thrusts: (1) communication and interface design for devices and data management, (2) reverse and forward engineering of neural systems and neural-inspired devices, and (3) control and adaptation technologies that express sensorimotor functions for individual needs.

The ERC/SNE will nurture future global multidisciplinary leaders. It will develop middle and high school project-based curricula that introduce neural engineering principles to students underrepresented in engineering. It will create multi-institution, undergraduate and graduate Neural Engineering courses with new degree structures and develop vertical research mentoring chains to build a strong research culture from faculty to K-12. It will build long-lasting and deep relationships through faculty and student exchange programs across all disciplines and partnering institutions, with a goal of removing barriers in communication across different fields, countries, and diverse backgrounds. The neural engineering field creates new pathways from the less quantitatively-based biological sciences to the more quantitatively-based engineering fields as well as pathways for people with disabilities to work in an engineering field that addresses their own experience and needs. The women and underrepresented minorities who currently account for over 40% of the Center's leadership team will serve as role models for students and starting faculty. Further, the ERC/SNE will extend its impact by identifying key technologies according to market significance and technical risk. The Center's portfolio will be constructed to deliver a steady stream of innovations over the near and long term. Its industry partnership structure includes not only small and large firms that will help shape Center IPs, but also hospitals and investment firms that will ground research activities to technologies that will truly assist people in need and steer future neural engineering market directions.

The ERC/NSE will strive to enhance the human experience both for persons with neurological disabilities and for the coming generation of global and diverse engineering innovators. The Center's seasoned, multi-disciplinary team will transform healthcare, manufacturing, and the educational infrastructure to guarantee neural engineering global leadership.