Alexander G. Hauptmann - US grants

The services being developed in this project will enable patrons of the future National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL) to efficiently locate video resources and meld them into compositions that effectively support particular objectives.

Video is a rich medium for communicating visual, time-dependent phenomena and for providing real-world footage capable of illustrating and motivating science and mathematics concepts. Vast collections of video have captured field studies and experiments, documented discoveries in space and throughout our planet, and recorded events in our world and in micro-environments not accessible to the human eye. However, despite their potential for use in educational settings, video resources are often discounted or overlooked by science, mathematics, engineering, and technology (SMET) educators and students. Barriers include:

-- Loss of investment made by authors of video compositions who blaze pathways through the video information space. These pathways remain unmapped, and hence undiscovered by other information foragers with similar requirements.

-- Lack of support to tailor video resources to specific needs.

-- Frustration in searching and browsing video, as much time is invested in viewing numerous video clips to gauge their relevance.

-- Inability to locate pertinent video material, due to insufficient indexing of its contents.

This project is exploring ways to overcome these barriers by capturing and managing the threads of video information access, use, and reuse within the NSDL. Specifically, the project is working on the following services:

-- Creation and organization of annotations for video compositions and information pathways, enabling a dynamic information repository where one's diligent work in producing a stellar video lesson plan can be recognized, rewarded, archived, and reused in future overlays of video information.

-- Support for composition of video lesson plans and multimedia essays from component clips meeting the time, message, and pedagogical requirements of the NSDL patron.

-- Explicit video annotation mechanisms, whereby NSDL patrons can access reviews and other commentary aligned and synchronized with video resources.

-- Implicit annotation mechanisms for video, allowing information retrieval schemes with relevance judgments based on access frequency and incorporation of video resources into derivative works.

-- Enhanced content-based video search functionality derived from the integration of speech recognition, language processing, and image processing automated techniques.

The project's focus on video complements the research of others focusing directly on the text or image domains. The project team is well-positioned to pursue this work, given their past accomplishments with the "Informedia" project (NSF Award Nos. 9411299 and 9817496).

This work will develop algorithms and systems enabling people
to query and communicate a synthesized record of
human experiences derived from individual perspectives captured
during selected personal and group activities. For this research,
an experience is defined through what you see, what you hear,
where you are, and associated sensor data and electronic communications.
The research will transform this record into a meaningful, accessible
information resource, available contemporaneously and retrospectively.
We will validate our vision with two societally relevant applications:
(1) providing memory aids as a personal prosthetic or behavioral
monitor for the elderly; and (2) coordinating emergency response
activity in disaster scenarios.

This project assumes that within ten years technology will be capable
of creating a continuously recorded, digital, high fidelity record of
a person's activities and observations in video form. This research
will prototype personal experience capture units to record audio, video,
location and sensory data, and electronic communications. Each constituent
unit captures, manages, secures and associates information from
its unique point of view. Each operates as a portable, interoperable,
information system, allowing search and retrieval by both its
human operator and remote collaborating systems. An individual
cannot see everything, nor remember everything that was seen or
heard. The integration of multiple points of view provides more
comprehensive coverage of an event, especially when coupled with support
for vastly improving the memory from each perspective. The research thus
enables the following technological advances:

* Enhanced memory for individuals from an intelligent assistant using an
automatically analyzed and fully indexed archive of captured personal
experiences.

* Coordination of distributed group activity, such as management of an
emergency response team in a disaster relief situation, utilizing multiple
synchronized streams of incoming observation data to construct a "collective
experience."

* Expertise synthesized across individuals and maintained over generations,
retrieved and summarized on demand to enable example-based training and
retrospective analysis.

* Understanding of privacy, security and other societal implications of
ubiquitous experience collection.

The foundation for this work, the Informedia Digital Video Library,
has demonstrated the successful application of speech, image, and
natural language processing in automatically creating a rich, indexed,
searchable multimedia information resource for broadcast-quality video.
The proposed work builds from these technologies, moving well beyond
a digital video library into new information spaces composed of
unedited personal experience video augmented with additional sensory
and position data. Tools will be created to analyze large amounts
of continuously captured digital experience data in order to extract
salient features, describe scenes and characterize events. The
research will address summarization and collaboration of multiple
simultaneous experiences integrated across time, space and people.

CareMedia provides automated video and sensor analysis for geriatric care. Through activity and environmental monitoring in a skilled nursing facility, a continuous, voluminous audio and video record is captured. Through work in information extraction, behavior analysis and synthesis, this record is transformed into an information asset whose efficient, secure presentation empowers geriatric care specialists with greater insights into problems, effectiveness of treatments, and determination of
environmental and social influences. CareMedia allows the behavior of senile dementia patients to be more accurately interpreted through intelligent browsing tools and filtered audiovisual evidence, leading to
treatment that reduces agitation while allowing awareness and responsiveness. The research begins with disruptive vocalization, a particular behavior noted across senile dementia assessment scales. The
coverage is then broadened ambitiously to integrate sensor and visual data for behavioral analysis and summarization in support of OBRA regulations requiring behavior management strategies that are not just chemical restraints. This effort includes automatic techniques to recognize disruptive vocalizations, more complex behavioral occurrences such as falls or physical aggression, and circadian patterns of activity. This research builds on key Carnegie Mellon research efforts in digital video analysis, wearable mobile computers, computer-based vision systems, and information retrieval systems for multimedia metadata.

A Video Indexing Ontology Using Fuzzy Metadata

Video has been very difficult to understand by a machine. While humans
have a seemingly "direct" way of seeing something and understanding a
scene in terms of background, foreground objects, and motions, video
understanding has been one of the perplexing problems of automatic video
and image analysis to date.

This project proposes a "divide and conquer" approach, where several
hundred general-purpose concepts (e.g., outdoors, animals) will be used
to describe and annotate a very large universe of scenes commonly
depicted in video. Analogously to a limited vocabulary set of indexing
terms one might find in a library card catalog, each video scene can be
annotated through a combination of these concepts ("metadata"). To go
beyond a mere listing of objects, actions and scenes visible in the
video, carefully chosen concepts also allow the description of
relationships between them ("ontology"), which allows for much richer
composite descriptions. The challenge will be to define these concepts
so that they satisfy several criteria at once:
* The concepts must represent things frequently visible in video
broadcasts.
* The concepts must be clearly identifiable to give computer
algorithms a chance to detect them automatically.
* The concepts must be linkable into an ontology that defines how
concepts are related.

Since video annotations, whether done by a computer or a human, always
will contain errors, this work will incorporate probabilistic confidence
metrics ("fuzzy metadata") into the annotation. No existing indexing and
classification schemes have explicitly defined standards for measuring
and reporting errors and omissions of indexing annotations; since
librarians and archivists have traditionally assumed that an index
contains only complete, trusted and verified metadata.

Furthermore, the project will assess to what extent these concepts can
be automatically extracted with state of the art video analysis
techniques. Using footage from documentaries and television news, the
project will perform video search and retrieval experiments to determine
the usefulness of the ontology and the confidence of the annotations.

URL: http://www.informedia.cs.cmu.edu/ontology

This interdisciplinary effort develops, integrates, and refines a suite of activity and behavioral observation tools supporting the automatic collection, annotation, access, analysis, and archiving of behavioral data for individuals and groups. These tools capture a continuous audiovisual record of human activity in various settings and apply machine intelligence technology to automatically process that record for efficient use by analytical observers to monitor situational behavior over time. The annotated record provides a level of completeness not feasible with human observers, allowing, for the first time, large-scale longitudinal behavioral and clinical research based on continuously captured and processed data, enabled through extensible interfaces accessing such voluminous records in a user-friendly, yet utilitarian manner. The record will be processed with data reduction and extraction technologies that recognize faces and speech, track moving individuals, and identify social interactions, while protecting the confidentiality of the participant's identity in collaboratively accessible, computerized databases.

The tools focus on the automatic identification of features within the audiovisual record to improve the accuracy and completeness of manual, labor-intensive rating instruments. Through computer vision, speech recognition, sensor integration, and machine learning, multimedia data extraction technologies will be developed for individual behavioral measurements. Additional tools will be developed to mine the resulting annotated longitudinal datasets for insights into individual interactions and reactions. The relevance of the tools will be demonstrated, refined, and validated through an initial challenge application and environment: the elderly residents in a continuing care retirement community. Collaborating studies of parent-child, teacher-student and autistic children social interactions will validate portability and utility across domains of behavioral research. The tools will be extensible, enabling behavioral scientists to better accommodate novel situations and source material. Specifically, the scientist will be able to identify a need for a class of audiovisual detection, adeptly supply training material for that class, and iteratively evaluate and improve the resulting automatic classification produced via machine learning and rule-based techniques.

The project's broader impacts include supplying social and behavioral scientists with automated tools supporting novel approaches to creating and analyzing data in their endeavors to characterize human behavior. Research by sociologists, psychologists, anthropologists, and medical clinical investigators will be enabled in new modes with greater accuracy and precision of observation than has heretofore been possible or practical. Ethical issues are explicitly and proactively addressed to engage the larger social concerns.

The video analysis community has long attempted to bridge the gap from low-level feature extraction to semantic understanding and retrieval. One important barrier impeding progress is the lack of infrastructure needed to construct useful semantic concept ontologies, building modules to extract features from the video, interpreting the semantics of what the video contains, and evaluating the tasks against benchmark truth data. To solve this fundamental problem, this project will create a shared community resource around large video collections, extracted features, video segmentation tools, scalable semantic concept lexicons with annotations, ontologies relating the concepts to each other, tools for annotation, learned models and complete software modules for automatically describing the video through concepts, and finally a benchmark set of user queries for video retrieval evaluation.

The resource will allow researchers to build their own image/video classifiers, test new low-level features, expand the concept ontology, and explore higher level search services, etc., without having to redevelop several person year?s worth of infrastructure.
Using this tool suite and reference implementation, researchers can quickly customize concept ontologies and classifiers for diverse subdomains.

The contribution of the proposed work lies in the development of a large number of critical research resources for digital video analysis and searching. The modular architecture of the proposed resources provides great flexibility in adding new ontologies and testing new analytics components developed by other researchers in different domains. The use of large diverse standardized video datasets and well-defined benchmark procedures ensures a rigorous process to assess scientific progress.

The results will facilitate rapid exploration of new ideas and solutions, contributing to advancements of major societal interest, such as next-generation media search and security.

URL: http://www.informedia.cs.cmu.edu/analyticsLibrary

This research project will develop methods and algorithms to assist people with the procedures required to operate their home medical devices without errors. The goal is for the system to be trained on sample recordings of the person operating the device correctly, so that later it will be able to detect deviations from the correct operation sequence in the currently performed procedure, and automatically provide corrective feedback to the user, including segments of the video portions that show the appropriate steps. The project provides a new paradigm for learning by observation that does not require complete understanding of detailed activities in arbitrary visual and sensor sequences, but merely aligns a given new sequence in known context with previously established training data to detect significant deviations. The approach has four components: (1) defining the key states in an operational procedure and the sensors required to best detect and later communicate the proper operation of a portable home medical device; (2) training the system by observing multiple correct operations; (3) observing a new instance of the operation sequence and recognizing that this operation deviates from the training data in a significant way; and (4) providing corrective feedback to the user in the form of audio and video prompts. The research aims to understand the common types of steps required in the operations of home medical devices, map how the critical indicators of these steps can be detected through appropriate sensors, train a system to recognize these steps in the context of a specific human operator, establish a range of required training repetitions for different operational step types and corresponding sensors, and provide a set of suitable interventions to the end user when errors occur. The experiments will establish the range of training data set sizes for the automated classification of device operations. The research expects to yield a taxonomy of typical operational steps from an observational perspective for a set of devices such as infusion pumps, and establish the most effective sensors or sensor combinations to detect the successful completion of each type of step. In addition, the work will help find suitable passages in the video portion of the training observations to use as corrective feedback, together with other interactive dialog interventions that may be appropriate for the particular user.

The long-term goal of this research is to develop a cognitive assistance system to learn and represent sequences of steps in the operation of home medical devices through multi-sensor observation and interaction with a human operator. Examples of some of the targeted home healthcare devices are respirators and nebulizers (to help breathing), dialysis machines, infusion pumps, home monitoring devices for blood pulse oxygen, EEG, and ECG. The project will develop a means for home users of these devices to ensure that the correct procedures are followed and accurate operations result. The system provides ongoing feedback to assist users in their device operation by ?watching? the process via different sensing technologies and providing appropriate guidance when required. The target population immediately benefiting from this work would be patients with mild cognitive impairments who would be supported with the automated coach in their use of home medical devices. The growing user base includes elderly people living at home, but requiring support from home medical devices. These medical devices can allow a patient to live independently with minimal assistance, as long as the home medical devices provide the required health support. The end result may be a reduction in errors and in the number of calls for assistance in the operation and maintenance of home medical devices. This will allow people to live independently at home for an average longer period than at present and thereby reduce health care system costs. The research is valuable for medical device companies with respect to device design, verification, and validation processes, offering insights into what sensors and communication devices could be most beneficial for integration into the device itself.

DESCRIPTION (provided by applicant): This application addresses broad Challenge Area (01): Behavior, Behavioral Change, and Prevention, and specific Challenge Topic 03-MH-101: Biomarkers in Mental Disorders. Clinically significant depressive symptoms have been reported in up to 86% of nursing home residents [Brodaty01], with substantial under-recognition of the problem and excess morbidity and mortality in individuals with dementia. The sensitivity of current depression detection approaches deteriorates notably with progressive cognitive impairment due to: (i) diminishing ability of demented subjects to communicate affective states, (ii) critical overlap between the neurovegetative symptoms of depression and the neuropsychiatric features of dementia (e.g., apathy), and (iii) the transient and fluctuating nature of depressive symptoms in the context of dementia [Bielinski06]. In order to improve the assessment of depression in dementia patients, the NIMH has proposed modifications to the DSM-IV criteria for major depressive disorder that 1) highlight the transient and fluctuating nature of depressive symptoms, 2) addition of social withdrawal and irritability items, and 3) substitution of the anhedonia criterion with "decreased positive affect or pleasure in response to social contact and usual activities" [Olin02]. As the subject's ability to report internal affective states diminishes, however, activity and behavioral correlates of depression such as verbal and motor agitation, physical aggression, care resistance, and food/fluid refusal become important markers of possible depression [Bielinski06]. Emerging video, audio and sensor technologies hold promise for quantifying and automating the detection of such activities and behaviors, and thus identifying stages of depression that may be differentially responsive to various prevention and intervention strategies. We propose to overcome the under-diagnosis and failed recognition of symptoms of depression in dementia by applying real-time continuous video and audio recordings in the non-private spaces of a nursing home special care dementia unit (SCU;16 total beds), as well as sensor recordings (radiofrequency identification tags, motion, contact and pressure sensors) throughout all the SCU spaces to capture the consenting subjects'activities, behaviors and social interaction patterns. Current "gold standard" research rating instruments will also be administered to record a detailed account of the cohort's clinical characteristics. The recordings will take place in three two-week waves that are each separated by 3 months. A frame-by-frame annotation of the video recordings by trained coders, as well as the pattern of sensor recordings, will provide training data for computer-based machine learning algorithms that will automate the detection of potential activity and behavioral manifestations to distinguish depressed from non-depressed nursing home residents with cognitive impairment or dementia. PUBLIC HEALTH RELEVANCE: The technology developed and applied here may ultimately lead to automated assistance in elder care through more complete and accurate observational records for depression diagnosis and evaluation than possible now with limited staff in long-term care facilities and for elders living alone at home.

This research addresses interactivity and scalability in automatically analyzing large collections of video and multiple video streams processed continuously. This work is developing mechanisms to enable real-time interactive video search for user defined concepts using intelligent, active processing clusters and methods for performing high-accuracy semantic video analysis from large amounts of weakly-labeled video over distributed computing resources. The methods leverage modern cluster file systems where data is stored on the local disks of the compute servers, and the location of data is made available to the runtime system to allow co-location of compution and storage.

The specific research objectives are to allow co-location of compute and storage through a runtime for parallel stream processing that parallelizes data processing and machine learning tasks across a cluster of multi-core compute nodes. The project also extends distributed versions of graphic model algorithms to speed computation of both the basic low-level signal processing steps and for the semantic analysis based on weakly labeled video data as currently available on the web. The main outcome is to demonstrate vastly accelerated, complete processing of parallel live video streams into a retrieval database with immediate search capabilities and accessing cluster resources during interactive search. The goal of this work is to develop principles for interactive applications driven by real-time processing of high-rate streaming data. The processing architecture and modules developed in this work will enable computer vision and multimedia developers to efficiently apply and test their own methods within this framework.

This research seeks to develop novel machine learning algorithms that enable real-time video and sensor data analysis on large data streams given limited computational resources. The work focuses on healthcare as an application domain where real-time video analysis can prevent user-errors in operating medical devices or provide immediate alerts to caregivers about dangerous situations. The research will develop algorithms to automatically adapt data analysis approaches to maximize accuracy of analysis within a short time period despite limited available computing resources. Today's healthcare environment is significantly more technologically sophisticated than ever before. Many medical devices are now frequently used in patient's homes, ranging from simple equipment such as canes and wheelchairs to sophisticated items such as glucose meters, ambulatory infusion pumps and laptop-sized ventilators. The rapidly growing home health industry raises new safety concerns about devices being used inappropriately in the home setting. The proposed research is designed to reduce medical device related use-errors by developing computational algorithms that perform real-time video analysis and alert the patient or caregiver when medical devices are not used appropriately. The real-time video and sensor data analysis is also critical to the healthcare systems that monitor the activities of the elderly or those with disabilities in order to allow a caregiver to react immediately to an incident.

New machine learning theories and algorithms will automatically adapt to hardware limitations, with the aim to learn from a large number of training examples, a prediction function that (i) is sufficiently accurate in making effective predictions and (ii) can be run efficiently on a specified computer system to deliver time critical results. Three types of prediction models are studied to address the problem of automatic hardware adaptation, including a vector-based model, a matrix-based model, and a prediction model based on a function from a Reproducing Kernel Hilbert Space (RKHS). A general framework and multiple optimization techniques are being developed to learn accurate prediction models that match limited memory and computational capacity. The new learning algorithms will be evaluated in several medical scenarios through real-time prediction of a patient's activities from observations in the large video archives collected by several healthcare related projects. The intellectual merit of the proposed work is in bridging the gap between the high complexity of a prediction model and limited computational resources, a scenario that is encountered in many application domains besides healthcare. The proposed research in machine learning algorithms and theories will make it possible to run complicated prediction algorithms on big data within the limitation of a given computing infrastructure. The developed techniques for automatic hardware adaptation will be applied to a large dataset of continuous video and sensor recordings for medically-critical activity recognition. The project's broader impacts include providing medical experts with algorithms and tools supporting novel approaches to analyzing observational data in their quest to recognize and characterize human behavior. Surveillance systems with continuous observations will be able to categorize salient events with co-located, limited hardware. Researchers with complex data from continuous streams will be able to explore their domains with greater accuracy within constrained time using their available computing resources. Similarly, large archives can be exploited as rapidly as possible with limited hardware.

This project aims to determine if the human brain has sufficient self-adaptivity, called plasticity, to operate brain-controlled robotic devices in performing tasks that require continuous coordination and synchronization with the person's natural limbs. These tasks are those where the action of one hand (or limb) depends on that of the other, here extended to include a robotic device, such as an unattached artificial "third hand." Besides advanced human prosthetics, application domains include co-robots as aides supporting emergency responders in hazardous environments performing complex manual operations in the field, in space and undersea, particularly where they need to perform independently without the assistance of other persons. The results are potentially transformative for human-computer interaction and cyber-human systems research and applications.

The research performs a set of non-invasive human subject experiments to determine whether the human brain may have sufficient neuroplasticity to enable "asymmetric dependent" operation of a detached robotic "third hand" (those in which the action of one hand depends on that of the other) through a brain-machine interface, or whether it may be limited in that capability by 2.5 million years of hominin evolution of our limbs and their extremities. This is a critical consideration as we explore the many issues of augmenting human physical capability with control implemented through brain-machine technology. Designs combining varying degrees of robot autonomy with human-machine verbal and gesture communicated control are also considered as supplements or alternatives to direct brain-control.

Human communication and information consumption is multimodal by nature, involving rich data and interaction of various modalities such as text, sound, vision, touch, and more. The field of multimedia has been responsible for developing theoretical foundations, novel methodologies, and effective system designs for integrating data of heterogeneous modalities to extract useful information, enhance interaction, and deliver optimal user experiences in different multimedia applications. Today the use of advanced multimedia solutions is pervasive in myriad applications and platforms such as social media, virtual/augmented reality, distributive collaboration and communication, online education, health care, scientific discovery, and many others. Now with the explosive deployment of multimedia devices, processes, and systems, the role of multimedia research has become even more important than ever, influencing our abilities and prospects in advancing state-of-the-art technologies and solving real-world problems underlying various challenges facing the society and the nation in areas like global communication, security, environmental sustainability, health care, education, etc. To respond to these challenges and further advance the frontiers of the field of multimedia, this project aims to develop an ambitious vision and concrete strategic action plans to guide the research directions and plan the critical resources needed in the multimedia field in the next ten years.

The project will organize a two-day workshop, in which invited leading researchers and practitioners from academics, industry, and research institutes will work together to brainstorm, discuss, and plan the most important research topics and directions for the multimedia field in the next ten years. Participants will be asked to provide assessment of the state of the art today, share best practices of organizing large-scale collaborative multimedia research, point out weakness and missing solutions and expertise in the community, and identify priority topics and directions that require further efforts and investment by the community and the funding agencies. Other important topics such as taskforce development, university-industry interaction, entrepreneurial translation, cross-disciplinary collaboration, and international collaboration will also be discussed. The outcome of the workshop will include a report that integrates input from all participants and conclusions from the workshop. The report and the input shared by the participants will be available for public access on the workshop web site. The workshop report will be disseminated through the mailing lists of the multimedia groups of ACM and IEEE and other professional communities, and will be included in the ACM Digital Library for archival and open access.