John Aloimonos - US grants

This first-year Presidential Young Investigator Award will support Dr. Aloimonos' work in active vision and navigational reasoning. His principle goal is to build robotic systems with robust, real-time visual capabilities for accomplishing specific navigational tasks such as moving object detection by a moving observer, object tracking, and obstacle avoidance. Dr. Aloimonos also intends to continue his research in robust scene analysis, visual learning, parallel computation, path planning, and other related topics.

This award is for the purchase of a multiple degree of freedom, high precision, light weight stereo camera head. The same equipment is being purchased at three other institutions in a shared research effort. Research topics include gaze control and target tracking, stereo and motion analysis, landmark-based navigation, automatic acquisition of object and environmental models, hand-eye coordination, dextrous manipulation with multi-fingered hands, real-time perception and manipulation. System software, algorithms and subsystems will be shared across the institutions. The University of Maryland will be purchasing equipment to support a multi-institutional shared research effort in active vision for robot navigation and manipulation. Other members of the consortium are the University of Rochester, University of Pennsylvania and the University of Massachusetts. Each institution will acquire the same state- of-the-art binocular camera head of high precision and high speed, and will share in the development of systems software and application software.

The PI will investigate the stucture and function of new multiview eyes, that is new imaging devices built out of several conventional cameras. The research is motivated by the variety of eye designs
in the biological world and obtains inspiration for an ensemble of computational studies that relate how a system sees to what that system does. This, coupled with the geometry and statistics of multiple views, points to new ways of constructing powerful imaging devices which suit particular tasks in robotics, visualization, video processing and augmented reality. First, the "Argus" eye will be designed and built, which is a system
consisting of up to 12 cameras attached to a sphere-like structure and pointing outwards. The video cameras, equipped with a synchronizer will simultaneously record data onto disk. Next new algorithms will be developed and implemented for processing video collected by the new sensor for the purpose of very accurately estimating 3D motion and subsequently shape models of the imaged scene. how to use such models in video editing and manipulation will be shown. In the second stage of the project, new algorithms for "negative spherical eyes", i.e. networks of video cameras attached to the walls of a room pointing inwards, will be developed. The main effort will consist of algorithms for recovering descriptions of action, in particular human movement. Specifically, software will be developed for building representations of 3D motion fields, and these structures will be analyzed using geometric and statistical techniques to develop invariant characterizations of action. It is expected that these developments
will have a strong impact to automatic animation.

The project will develop an integrated approach to decision making under uncertainty based on robust extraction of probabilistic spatio-temporal information from sensor network data, efficient management of collections of such information, dynamic situation assessment with the aid of these collections, and representations of decision problems about these situations in terms of partially observable Markov decision processes. This work will bring together recent advances in the fields of visual sensor processing and database technology for management of uncertain information.

The PIs will undertake an experimental study and computational modeling of the internal representations and associated processes that underlie action perception and understanding by observers, and action planning and execution by actors. To facilitate both careful experimentation and formal theory, the PIs will approach the behavior representation problem primarily through the visual system, asking how do we understand the actions of others using our vision? That is, how do we perform mappings from image sequences depicting simple actions to the corresponding internal representations that allow action recognition, imitation, etc? The PIs will further explore higher-level cognitive representations and mechanisms used to categorize, reason about, and judge the movements and actions of others. The approach is based on a novel formal theory of the mental representations and processes subserving action understanding and planning, which the PIs believe provides a compact but powerful and extensible computational approach to the analysis and synthesis of complex actions (and action sequences) based on a very small set of atomic postural elements ("key frames" or "anchors") and the corresponding probabilistic, grammatical rules for their combination. This probabilistic "pose grammar" approach to action representation is similar to state of the art techniques used for speech recognition (e.g., hidden Markov models), but with key postural silhouettes taking the place of phonemes; such augmented transition grammars also nicely reflect sophisticated new control-theoretic techniques in robotics for robust anthropomorphic movement. The action representational system is not monolithic, but rather occupies a spectrum of informational structures at hierarchical levels corresponding to different behavior "spaces": mechatronic space, used in movement planning and production; cognitive space, involving representations for action recognition, analysis, and evaluation; visual motion space, which encodes and organizes visual motion caused by human action; and linguistic motion space, comprised of conceptual/symbolic action encoding. Excluding here the latter space, the PIs' theoretic, computational, and experimental efforts seek to clarify and formally describe both the nature of the representations in these spaces and, crucially, the mapping of representations across spaces. Notably, they explore a candidate action representation, referred to as a visuo-motor representation, which, in facilitating the understanding of observed actions, may recapitulate and resonate with the actual motor representations used to generate movement. Moreover, they present a promising approach for obtaining this representation from discrete action elements or anchors.

Broader Impacts: This project will lead to significant advancements in both research and applications in psychology (e.g., robust social judgments given degraded biological motion), kinesiology (e.g., analysis/modeling/training of movement profiles, as in athletics or pathology/rehabilitation), robotics (e.g., control of anthropomorphic robots), human and computer vision (e.g., automated action recognition in digital video), and other fields concerned with the interpretation and production of human/humanoid action.

The goal of this project is to parse video from a moving platform in real-time to produce retinotopic maps that reveal the spatial layout of the scene as well as any independently moving objects present. The project proposes to duplicate the function of the early visual system in a multichip neuromorphic system with two hundred and thirty thousand silicon neurons (three quarters the number of pixels in a VGA image) and three billion synaptic interactions several orders of magnitude larger than anything built to date. The outcome of this project will be SEER, a subwatt, paperback-size seeing machine.

The intellectual merit of the proposed research stems from the tight synergy between the
computational theory, based on the principle of compositionality ,and the neuromorphic implementation, based on reentrant networks. Compositionality dictates that the various parts of the vision problem should be attacked simultaneously and reentrancy gives us the capability to do exactly that.

The broader impact of the proposed effort could be enormous. A multichip neuromorphic system performing multimodal segmentations would reinvigorate robotics and computer vision. By providing the infrastructure for early vision, it will facilitate the study of cognition, which will most likely generate a flood of new theories and experiments, in the Neurosciences and in the sciences of the artificial.

DESCRIPTION (provided by applicant): Human movement has been a large window into the functioning of the nervous system. Behavioral scientists have had major accomplishments, such as documenting movement milestones in human development and establishing a relationship between brain and behavior in typical and atypical populations. These measurements are performed today with a cornucopia of sophisticated techniques, ranging from infrared and video to wireless sensor networks. However, despite the tremendous progress on measuring human movement, we still don't fully understand, for example, motor decline in elderly people or Parkinson's disease during daily living activities at home and the workplace;or how atypical social interaction in autism or developmental coordination disorder are manifested in body gestures. Why can't we yet deal with problems of such nature? It is clear that the problems mentioned above have characteristics that are beyond the state of the art or any single discipline. Thus, we propose a novel, interdisciplinary, and multi-level motion understanding tool to extract multi-scaled, nested representations of transitive and intransitive actions and communicative actions at different levels of abstraction at the "individual" and "workgroup" levels. Our specific aim is to develop a Human Action Language (HAL) tool, a tool for describing and understanding human actions. The underlying premise is that the space of human actions is characterized by a language;this new language has its own phonemes (primitives), its own morphemes (words/actions) and its own syntax, semantics and pragmatics. Although previous research has concentrated on finding primitives in very often isolated types of human action, the innovation here is the use of large amounts of human motion data in ecologically valid settings and in conjunction with modern data mining and grammatical induction techniques. To validate the HAL tool, we will apply it to assess atypical movement in Developmental Coordination Disorder (DCD) and Parkinson's disease (PD). Specifically, we propose to extract the DCD grammar and the PD grammar and compare them with the grammars from the control populations, investigating relationships between the corresponding grammars at the individual and workgroup levels. Our interdisciplinary team consists of a computational scientist, a behavioral scientist (motor development) , and a computational neuroscientist (motor control and learning). The proposed tool will extend the scope of behavioral sciences (grounding of language, imitation, and gesture-based social communication) and facilitate interdisciplinary research bringing together movement disorders specialists, behavioral scientists, physical or occupational therapists and computer scientists. Several NIH Institutes would benefit from the availability of such a tool, including NIA/NINDS, NIMH and NICHD. The ultimate goal is to better understand human action production and understanding, and to develop optimal diagnostic and intervention tools for populations with atypical movement patterns. The proposed tool will extend the scope of behavioral sciences (grounding of language, imitation, and gesture- based social communication) and facilitate interdisciplinary research bringing together movement disorders specialists, behavioral scientists, physical or occupational therapists and computer scientists. Several NIH Institutes would benefit from the availability of such a tool, including NIA/NINDS - for understanding motor decline in the elderly and neurological populations in single and group-based daily living activities, NIMH - for understanding stereotypical behaviors in populations affected with mental disorders, and NICHD - for understanding developmental aspects of cognitive motor behavior in children at school or home. The ultimate goal is to better understand human action production and understanding, and developing optimal diagnostic and intervention tools for populations with atypical movement patterns.

The BehaviorScope project seeks to develop a framework for understanding patterns and behaviors from sensor data and metadata in distributed multimodal sensor nodes. Patterns and behaviors (especially of humans) will be parsed by a hierarchy of probabilistic grammars and other mechanisms into a compact and more descriptive semantic form. These higher-level interpretations of the data will provide the necessary network cognition needed to provide services in many everyday life applications such as assisted living, workplace safety, security, entertainment and more. The project will use a lightweight camera sensor network as its primary platform and will focus on two types of spatio-temporal data processing. At the local sensor's field of view, this research will investigate the design of filters for robustly detecting humans as well as their gestures and postures. At a more macroscopic level, collections of sensors will coordinate to detect longer term patterns of behavior. The expected outcome is a new data interpretation framework that can understand the spatial and temporal aspects of data and respond to them with meaningful services. To collect real data and to demonstrate the developed concepts in practical applications, this work will use assisted living as the driver application. In this context, the developed sensor network will supervise the behaviors of elders living alone at home to generate daily activity summaries, post warnings and alarms when they engage in dangerous activities, and provide a variety of services that increase the autonomy and independence of these individuals.

We propose to organize a two-day workshop on Visual Navigation with invited speakers covering a broad spectrum of scientific disciplines with emphasis on visual navigation. The goal of the workshop would be to assess the state-of-the art in this field and to critically examine future trends. Through presentations and panel discussions, our goal would be to identify a number of new research questions surrounding Visual Navigation that have intellectual merit and whose answers have the potential for broad impact. New ideas have the potential to come from a number of new developments.

One such development is the discovery of mirror neurons. This opened new avenues because it means that the motor system of the agent is an important space that is directly related to perception. Visual navigation involves movement of the agent. When an agent A observes another agent B perform a task, then agent A "understands" what agent B is doing because he transfers the movement to his own motor system. In other words, agents possess representations of their own actions at an abstract level. Since this abstract level is related to language, it is possible to envision navigating systems of the future that provide language descriptions of their environment. Another important development is what is broadly known in some circles as "Hyper-empiricism". The availability of gargantuan amounts of data facilitates the "learning" of a number of functions related to navigation. Lots of interesting questions arise about what to learn and how, as existing computational learning techniques are inadequate. Other developments we hope will show up at the meeting.

CPS:Small: Methods and Tools: Robots with vision that find objects

The objective of this research is the development of methods and software that will allow robots to detect and localize objects using Active Vision and develop descriptions of their visual appearance in terms of shape primitives. The approach is bio inspired and consists of three novel components. First, the robot will actively search the space of interest using an attention mechanism consisting of filters tuned to the appearance of objects. Second, an anthropomorphic segmentation mechanism will be used. The robot will fixate at a point within the attended area and segment the surface containing the fixation point, using contours and depth information from motion and stereo. Finally, a description of the segmented object, in terms of the contours of its visible surfaces and a qualitative description of their 3D shape will be developed.
The intellectual merit of the proposed approach comes from the bio-inspired design and the interaction of visual learning with advanced behavior. The availability of filters will allow the triggering of contextual models that work in a top-down fashion meeting at some point the bottom-up low-level processes. Thus, the approach defines, for the first time, the meeting point where perception happens.
The broader impacts of the proposed effort stem from the general usability of the proposed components. Adding top-down attention and segmentation capabilities to robots that can navigate and manipulate, will enable many technologies, for example household robots or assistive robots for the care of the elders, or robots in manufacturing, space exploration and education.

Cyber-physical systems of the near future will collaborate with humans. Such cognitive systems will need to understand what the humans are doing. They will need to interpret human action in real-time and predict the humans' immediate intention in complex, noisy and cluttered environments. This proposal puts forward a new architecture for cognitive cyber-physical systems that can understand complex human activities, and focuses specifically on manipulation activities. The proposed architecture, motivated by biological perception and control, consists of three layers. At the bottom layer are vision processes that detect, recognize and track humans, their body parts, objects, tools, and object geometry. The middle layer contains symbolic models of the human activity, and it assembles through a grammatical description the recognized signal components of the previous layer into a representation of the ongoing activity. Finally, at the top layer is the cognitive control, which decides which parts of the scene will be processed next and which algorithms will be applied where. It modulates the vision processes by fetching additional knowledge when needed, and directs the attention by controlling the active vision system to direct its sensors to specific places. Thus, the bottom layer is the perception, the middle layer is the cognition, and the top layer is the control. All layers have access to a knowledge base, built in offline processes, which contains the semantics about the actions.

The feasibility of the approach will be demonstrated through the development of a smart manufacturing system, called MONA LISA, which assists humans in assembly tasks. This system will monitor humans as they perform assembly task. It will recognize the assembly action and determine whether it is correct and will communicate to the human possible errors and suggest ways to proceed. The system will have advanced visual sensing and perception; action understanding grounded in robotics and human studies; semantic and procedural-like memory and reasoning, and a control module linking high-level reasoning and low-level perception for real time, reactive and proactive engagement with the human assembler.

The proposed work will bring new tools and methodology to the areas of sensor networks and robotics and is applicable, besides smart manufacturing, to a large variety of sectors and applications. Being able to analyze human behavior using vision sensors will have impact on many sectors, ranging from healthcare and advanced driver assistance to human robot collaboration. The project will also catalyze K-12 outreach, new courseware (undergraduate and graduate), publication and open-source software.