David H. Laidlaw - US grants

The impact of this work will be threefold. The immediate result will be a prototype system for using software visualization for understanding. Educationally, the work will expose students to visualization as a means of understanding and get them interested and involved in working on the difficult problems inherent in visualization and understanding. Finally, the broader impact of this project will be to establish foundations for future software visualization and understanding efforts by solving some of the basic problems in these areas.

Visualization of Multi-valued Scientific Data: Applying Ideas from Art and Perceptual Psychology

This is a multi-disciplinary research project to discover new visualization tools for interacting with and understanding multi-valued volumes of scientific data and the physical phenomena they measure. The tools will be developed and evaluated in close collaboration with scientists in three disciplines: neurobiologists studying neural development and disease via biological imaging, computational flow researchers studying blood flow through arteries, and geographers using remote sensing for environmental monitoring and resource management. We will factor out common patterns from the problems in these multiple disciplines to develop interaction metaphors and visualization techniques that are generalizable and widely applicable.

This project develops new visualization evaluation methodologies, an area that has only begun to be addressed. And it compares the effectiveness of visualization applications in several interactive and static computing and display environments including a 4-wall Cave, a 40'x40' virtual environment with a head mounted display, stereo head-tracked workbenches, desktop workstations, paper, and 3D rapid-prototyping output. Immersive environments will be studied
because the value of these non-traditional working environments has not been established and because they present an opportunity to explore fundamentally different interaction metaphors. Comparisons will be performed for both interactive and static cases with appropriate technology determined for each application.

This project brings together experience from art and perceptual psychology for inspiration. Through several centuries, artists have evolved a tradition of techniques to create visual representations for particular communication goals. Art history provides a language for understanding that knowledge. We will draw inspiration from painting, sculpture, drawing, and graphic design and apply these techniques to the scientific problems.

Beyond inspiration, perceptual psychology also brings a second set of knowledge to bear on scientific visualization problems. Evaluating the effectiveness of visualization methods is difficult because, not only are the goals difficult to define and codify, tests that evaluate them meaningfully are difficult to design and execute. These evaluations are akin to evaluating how the human perceptual system works. Perceptual psychologists have been developing experiments for understanding perception for decades, and they will help develop methodology and expertise for evaluating visualization methods in close collaboration with biologists, fluids researchers, geographers, artists, and computer scientists.

While many of the individual components of this project are important alone, the collaborative aspects are the most notable. Mining ideas from art and perception will suggest unusually innovative visualization ideas. The application of new visualization techniques and collaboration with researchers in other fields will provide us with a unique opportunity to validate the techniques and ensure that they are responsive to the needs of the scientific problems. Because the techniques will be developed with application to multiple disciplines, they are likely to find further application within these and other disciplines. The assembled team brings strengths in all of the disciplines and has already demonstrated a track record of collaborative work.

The broader impact of the proposed research lies not only in the information technology arena, where new methods will help scientists in many disciplines to more effectively interact with and understand their data and gain insight about the physical phenomena they represent, but also in the specific scientific domains we will study. The study of blood flow could lead to improved understanding of and treatment for cardiovascular pathologies. An understanding of early neural development could enable new therapies for birth defects, genetic disorders, and other diseases. Remote sensing advances could provide more effective resource monitoring and permit widespread improvements in global quality of life.

Much of human intelligence can be broadly characterized as the ability to identify patterns and the visual system is the most sophisticated pattern finding mechanism that we have. Of all of our Perceptual systems, vision dominates. It is estimated to engage 50% of the cortex and 70% of all our sensory receptors are visual, but is only just becoming possible to display as much information as the human visual system is capable of absorbing.
The proposed research has the following five elements: a) A display that supports real-time animation at the limit of the resolution of the human visual system, b) The development and evaluation of perceptually near-optimal solutions through the development of human-in-the-loop optimization techniques, c) A set of experiments that will measure the efficiency of different ways of displaying common information structures, including paths in graphs, aspects of flow patterns, and the shapes of overlapping surfaces, d) The development of algorithms that support the mapping between data attributes and visual display primitives in a flexible adaptive or tunable process, e) The application of the techniques to visualization problems in three areas: flow visualization, overlaying surface visualization and large network visualization.

The intellectual merit of the proposed research will be to firmly establish information psychophysics and define the field as an intellectual endeavor thereby combining existing techniques into a cohesive discipline. The high- resolution display will enable researchers to work at the limits of human perceptual capability endowing the results with long-term value.

The broader impacts of the proposed research will include design guidelines, immediately useable design solutions, as well as algorithms and information display theory. Material contributions will be made in the following areas: flow visualization, network visualization, and overlapping surface visualization.

With support from a National Science Foundation Major Research Instrumentation Award, Brown University will acquire a 3 Tesla magnetic resonance imaging (MRI) system. The MRI system will become housed in a research-dedicated MRI suite within the newly constructed Life Sciences Building at Brown, and it will form the core infrastructure for MRI-related research conducted by more than 100 faculty, research staff, and students in the Brown University community, including its College of Arts and Sciences and Medical School. Researchers at Brown will use the NSF-fund MRI system primarily to investigate fundamentals of brain structure and function. In addition to Brown users, researchers from other nearby institutions, such as the University of Rhode Island, Regina Saliva University, and University of Massachusetts-Dartmouth can have access to the 3 Tesla MRI system.

Non-invasive imaging of the human brain has become a key research tool for life scientists interested in understanding brain mechanisms of sensation, perception, cognition, and voluntary movement. MRI has becoming a cornerstone of such activities since it can provide structure and functional information at previously unobtainable brain locations without the need for invasive measures. Structural MRI can provide sub-millimeter resolution of the cellular and fiber tract regions of the brain. These capabilities now allow precise measurement of local brain volumes and visualization of the source and destination of major axon pathways. Functional MRI can rapidly measure local changes in blood dynamics in volumes as small as 1 cubic mm. Blood dynamics reflect changes in local neural activity, and its exploitation has become a key tool in exploring brain mechanisms of a variety of functions that constitute everyday experience. The NSF funded resource will allow Brown researchers and those from nearby institutions to develop new strategies and knowledge about how the human brain mediates complex behavior.

Projects currently planned for the 3 Tesla MRI system include research in systems and cognitive neuroscience and biomechanics. A major effort will be to enhance spatial and temporal resolution of structural and functional MR imaging, using special equipment of the new 3 Tesla MRI system. In particular, the infrastructure will facilitate investigating specialization of the myriad brain areas that process visual stimuli, not only across the brain, but also within each area. The new MRI system will allow non-invasive imaging of the input and output processing zones of cortical areas. Several investigators will interrogate the functional MRI signals obtained during single instances of perceptual experience or voluntary movement to predict the conscious experience of the observer or to predict the performed movement(s). While these 'mind-reading' efforts currently occur off-line, the team plans to implement them in real-time, and ultimately at high spatial resolution. The addition of the 3 Tesla MRI resource at Brown will boost ongoing educational and research activities and will stimulate novel interactions between students, faculty and researchers working across life, social, physical and applied science disciplines. The instrumentation will also be used in out-reach programs for under-represented minority high-school students participating in summer programs at Brown.

The research project is aimed to create a Dynamic Data Driven Application aimed at improving the understanding of a complex biophysical system - the flight of bats. The system is comprised of a multi-level hierarchical simulation of bat flight based on parameterized features of bat morphology and behavior. The simulation operates at multiple levels of physical approximation and computational speed, and is capable of very rapid solutions but requires input from measurements to ensure fidelity and optimality. This input will be provided in an integrated fashion, drawing from a parallel series of experiments in which several discrete data streams will be generated, including kinematic wing data, wake velocity data over a series of two dimensional cutting planes, as well as other data such as bone deformation, experimentally-determined material properties, etc. This data will direct the simulation ensuring accurate solutions, but still with high responsiveness. The data streams will be monitored, synthesized, combined and processed using an advanced immersive visualization environment which will be used to guide the interactions between the measurement and simulation and to organize the disparate streams of data. The simulation environment to be developed is the first such system capable of generating timely solutions of complex flows over highly unsteady and deformable structures. This has multiple scientific and engineering benefits ranging from the ability to address fundamental questions in evolutionary biology to the design of bio-mimetic structures that draw from the abilities of bats on the wing. The direction provided by the experiment will guide scientists to the key sensitivities of these complex flying systems and provide insight to the complexities of animal flight mechanics. Lastly, the visualization systems will provide a unique tool for the synthesis and management of dynamic data drawn from a wide and disparate variety of data sources each having different qualities.

This award is for the development of instrumentation for 3D visualization of rapid skeletal motion in vertebrates. Its two primary components are (1) a high-speed, biplane X-ray fluoroscopy system and (2) automated software for precise, 3D skeletal animation by aligning 3D CT bone models with pairs of 2D X-ray images. The result will be a substantial advance over technology that is currently available for research in vertebrate functional morphology and biomechanics. The objective is to make dynamic 3D skeletal imaging an affordable and widely available technique. The new combination of high-speed, biplane X-ray and 3D visualization software is named "CTX imaging." Vertebrate functional morphology is an active and growing subfield of organismal biology in which the mechanical and evolutionary relationships between anatomical form and biomechanical function are investigated. For example, the action of long tendons as springs in kangaroo hopping, the effect of mouth size and shape on suction feeding in fish, and the function of the "wishbone" in bird flight have all been explained in the past two decades by functional morphologists. New discoveries in functional morphology have consistently been driven by the introduction of new technologies, such as high-speed cameras, electromyography, force plates and digital particle image velocimetry. Natural movements in animals almost always occur in 3D and often are very fast. Quantification of rapid skeletal movement in 3D would be a powerful technique for relating form to function, but functional morphologists have had no technique for measuring bone movements in 3D. The movement of external markers on the skin is generally used as a proxy for skeletal movement, but skin is often loose and the markers do not track the skeleton well. CTX analysis of a CT scan plus two X-ray movies will produce a highly accurate 3D animation of skeletal elements moving in space. These will be more than stick figures -- the complete 3D morphology of each bone will be present and animated precisely with this technique. Biplane X-ray imaging and CTX analysis will make it possible to study aspects of skeletal kinematics that are largely inaccessible with other techniques, such as long axis rotation of bones, putative bending of fine bones in small animals, and the relative 3D motions of the articular surfaces of joints. CTX will provide more accurate data for input into musculoskeletal models, such as joint angles for inverse dynamics and neural control models. This is an interdisciplinary proposal combining the expertise of two functional morphologists (Brainerd and Gatesy) who have extensive experience with dynamic X-ray imaging of animal movement and a computer scientist (Laidlaw) who specializes in building computational tools for accelerating science, with particular emphasis on scientific visualization tools.

Three-dimensional visualization of rapid skeletal motion in vertebrates will be possible with instrumentation to be developed under this award. CTX imaging, made possible with this instrumentation, will open up new areas of research in vertebrate functional morphology, such as the comparative study of 3D joint biomechanics. The animations developed with CTX will be powerful scientific tools, but they will also be accessible and appealing to the general public. These animations will be used to increase appreciation for basic research whenever possible.

Proposal #: CNS 09-23393
PI(s): Laidlaw, David H.; Hesthaven, Jan S.; Karnadiakis, George E.; van Dam, Andries
Institution: Brown University
Providence, RI 02912-9002
Title: MRI/Dev.: Next-Generation Interactive Virtual-Reality Display Environment for Science

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).



Project Proposed:
This project, developing a world-class interactive large-field-of-view 95 megapixel immersive virtual-reality environment, aims at creating a novel, demonstrably useful, rich, and expressive interaction, visualization, and analysis that truly leverage the human visual and motor systems in Virtual Reality (VR).
This work intends to help accelerate scientific work, research into innovative visualization methods for accelerating science in the future, and even leverage the fundamental advantages of immersive large-field-of-view visualization and body-centric human-computer interaction. Two decades of research have established the value of immersive displays as a research tool in many scientific domains and has also identified a set of currently unmet needs that block application of such displays to new problems and domains. These needs encompass high display resolution, brightness, contrast, and size; fast, responsive tracking with high accuracy and low latency; ease of use in working with new kinds of data; and reliability. Although a few multi-million dollar systems exist that may be able to address these needs, these few systems do not match the proposed display?s color gamut, small physical space requirement, and lower replication cost. The system is expected to support more natural and effective interaction with data than the current 3D point-and-click wand driven CAVEsTM by maximally utilizing as appropriate full-body, motion-captured user interactions and gestures. More display information will be made accessible to the human visual system with less user effort by matching, or exceeding the perceptual qualities of a modern LCD monitor. An immersive stereo display will be integrated with the perceptual resolution of a desktop display and superior brightness and contrast. Integration of software tools for creating virtual-reality applications quickly will address ease-of-use and reliability. The new tools are expected to be simple, support a spectrum of displays, and provide rich support for gestural interaction. A monitoring process to identify potential problems among the interacting hardware and software components will be put in place to identify and address problems before instruments are delayed. Users of the system include planetary geologists, systems biologists, brain scientists, cell and molecular biologists, biologists studying animal motion (including flight), fluid dynamicists, bioengineers studying arterial hemodynamics, visual designers developing interactive techniques for scientists, digital literary artists, and visualization and interaction researchers. Within interaction research, experiments using the system are expected to establish the appropriate level of display technology (e.g., resolution, interactivity, or stereographic display) needed for different classes of scientific analysis. The techniques, monitoring system, and software environment will be distributed on SourceForge to respectively help accelerate scientific progress nationwide, for developing multi-display applications, and for ensuring reliability.

Broader Impacts: While educating many students, the instrument is expected to enable new advances in all of the scientific disciplines of the users listed above, including a better understanding of the workings of cells and genes and proteins they contain (which could consequently improve quality of life broadly), behavior of fluids in arteries and around moving animals, animal locomotion (which could lead to improved biomimetic locomotive, floating, or flying vehicles), the wiring of the human brain, how it affects human capabilities, and how it can degrade; and Mars. The efforts are likely to produce a new generation of scientists who can better analyze research problems using scientific visualization, computer scientists more cognizant of scientists? analytical needs, and artists and designers who can accelerate the design process for immersive scientific visualization tools.

This collaborative research project (IIS-1320046, IIS-1319606) designs a 3-dimensional immersive visualization environment for volume data that is critical in a variety of application domains, such as medicine, engineering, geophysical exploration, and biomechanics. For example, biomechanics researchers examine volumes derived from insect scans to understand how form relates to function, particularly in regard to how insects create internal fluid flows. For effective analysis of a 3D volume, scientists and other users need to integrate various views, peer inside the volume, and separate various structures in the data. However, despite many advances in volume rendering algorithms, neither traditional displays nor traditional interaction techniques are sufficient for efficient and accurate analysis of complex volume datasets. This project develops an approach for interactively exploring and segmenting volume datasets by combining and extending: (1) utilization of advanced, high-fidelity displays based on virtual reality (VR) technologies for improving the visual analysis of volume data, and (2) the use of natural, gesture-based 3D techniques. Using controlled, empirical studies with real-world volume datasets from biomechanics and other biological sciences, the investigators are determining what characteristics of advanced displays can lead to faster, more accurate visual analysis. Iterative design and evaluation methods are being used to develop usable and natural 3D interaction techniques with which users can explore the interior of volume datasets. Beyond the empirical findings of these studies, an important outcome of the project is the design of a next-generation volume data analysis system that can be used by scientists and researchers to improve the efficiency and accuracy of their work.

The expected results will provide a deep understanding of visualization principles fostering further advancements in the realm of volume data analysis. Easier, more accurate, and faster analysis can lead to improvements in healthcare, breakthroughs in science, and advances in education. For example, this work may lead to insights into fundamental physiological mechanisms of feeding, breathing, and circulation in insects - one of the most important animal groups on earth. There are millions of insect species living in almost every habitat, and their lifestyles have profound impacts on human societies. Their effects in areas such as agriculture and health can be both positive (e.g., pollination) and negative (e.g., crop damage, disease), and understanding their fundamental physiologies is critical to controlling their impact. The project provides opportunities for interdisciplinary educational and research activities for graduate and undergraduate students, and outreach activities to underrepresented students. The results of this work will be disseminated broadly via publication in archival journals, peer-reviewed conferences, and online forums. The project website (https://research.cs.vt.edu/3di/node/188) will provide access to research results, including data sets and software.