Victoria Interrante - US grants

The objective of the proposed reserch is to develop a new class of data analysis methods. This will involve on-the-fly identification of physically-important events selectively store the data. This database will then be interrogated by feature extraction algorithms to yield an additional object database, which is a compact representation of the physically important space-time object trajectories. These database objects will be used as input to novel data mining methods to discover causal relationships between the objects. This approach will result in efficient storage of data, visualization of the important events within the data set, and methods for the high-level analysis of relationships between objects in the data.

Visualization - the visual representation of information through images -
is vital in a broad range of scientific and medical applications. A good
visualization conveys the essential information in a complex dataset
clearly and effectively, emphasizing important features and minimizing
extraneous detail. The usefulness of a particular visualization technique
can be quantified in terms of the extent to which it improves task
performance for a given application. This makes it possible to compare
different methods or to pick optimal parameter values for a given method.
The process of designing an effective visualization, however, remains for
the most part a black art, terribly ad hoc, largely guided by intuition.
The PI's research is a quest for a deeper understanding of the science
behind this art, guided by theoretical and experimental insights from human
visual perception. This can be used to enhance all computer interfaces
that use elements studied here.

Specifically, the focus of the research is to investigate
perceptually-based methods for more effectively conveying the
three-dimensional shape and relative depth of smoothly curving surfaces in
scientific datasets. Key applications for this work include the
visualization of superimposed isovalue surfaces in 3D scalar fields, and
the combined visualization of external and internal structures in medicine
and CAD. The investigator will begin by using controlled observer
experiments to thoroughly evaluate texture as a device for enhancing the
perception of shape. This will identify and quantify the key aspects of
texture in interface design. The investigator will also pursue the
automatic generation of non-photorealistic, pen-and-ink style
representations of arbitrary polygonally-defined objects for interactive applications.
The advantage of such techniques is that some datasets need to be represented in a more
subtly indicative style than traditional graphics renderings. Complementing
the research component of this project is an educational component,
directed at both undergraduate and graduate students, aimed at inspiring
interest and fostering creativity through hands-on experiences in
visualization.

This project, constructing a stereoscopic, high intensity, immersive projection environment for interactive design, supports collaborative research in visualization, perception, architectural design, and computer graphics. Through collaborative efforts of computer scientists and architects, the work seeks to develop and use tools that allow the intuitive creation, manipulation, and display of data an images within a stereoscopic, high ambient intensity, multi-user, immersive projection environment, to achieve fundamental breakthroughs that facilitate the creative design process, and enhance the understanding of science behind the art of effective visual representation. At the same time, new algorithms for effective interactive rendering and manipulation of large scanned environmental datasets and new tools for physically and perceptually realistic color appearance rendering will be developed. The infrastructure contributes to the following research:
Harnessing the Fullest Potential of Virtual Environment Technology for Research in Visualization and Design,
Developing New Tools for Conceptual Architectural Design in Immersive Virtual Environments,
New Tools for Efficient Rendering and Design Conceptualization, and
Computer Aided Color Appearance Design.
The first project aims at determining how to most effectively harness the special potential of virtual environment displays for data representation. Foremost will be efforts to investigate methods for facilitating and exploiting the expanded spatial understanding afforded by immersive displays, exploring the impact of employing a range of non-photorealistic representational styles in immersive virtual environments. The second builds on an immediate-mode modeling system and software, supporting the geometry creation and manipulation, and texture, image and video insertion and manipulation. Goals include offering the designer a space that supports designing (not just an empty space in which to stand); investigating the kind of contextual environment that best supports the design in its various stages; different aspects with different scales; finest expression of ideas; support of the supply and generation of information, text, image, video, and sound within the design environment; support of symbols and representation within the virtual environment; and determining those aspects of the design that are best done outside the environment and the best way to bridge the two worlds. The third develops a tool to allow modeling geometry on top of the finished sketch design, much like the experience of working with tracing paper. The last uses the panoramic display system to evaluate the results of computer aided color appearance design (CACAD) research that was performed on either cathode ray tubes or other specialized CACAD workstations. At the fundamental computer graphics level, this work is expected to expand the state of the art in virtual reality by evaluating the contribution that surface reflection makes to the illusion of "presence." The work explores whether expensive surface finish samples and controlled observing conditions can be replaced by computer graphics simulations and a "virtual color proofing" system.

The impact of the research will extend to new courses in CS and architecture. The team formed the Digital Design Consortium, an outreach effort involving community leaders.

In this project two PIs, one from Computer Science and the other from Architecture, will collaborate to investigate ways of effectively harnessing the full potential of immersive virtual environments technology to empower the creative processes involved in the earliest stages of architectural design. Through these efforts they expect to gain insight into how to enhance spatial awareness, facilitate spatial understanding, and enable the intuitive creation, manipulation, and comprehension of inexactly or incompletely specified geometric and environmental models, as well as abstract information, within immersive virtual environments that support the integrative nature of design. This project will lead to new and deeper understanding of issues relating to design research and education, and especially into how visual presentation, information representation, and user interaction affect spatial awareness and understanding, with architectural design as the driving application. Another project outcome will be new tools, in particular a virtual environment system suitable for the support of the activity of design, including new methods for content creation, new methods of information integration, and new algorithms for rendering a range of different kinds of models in a variety of realistic and non-photorealistic styles, with the objective of efficiently and effectively portraying large, complicated architectural datasets, including both building details and the accompanying environmental surround, in a perceptually effective manner that can capture multiple levels of ambiguity in the architectural specification. In addition, a diverse series of formal studies will be conducted to gather information about specific characteristics of the design process, including the efficiency and effectiveness of the interpretation of 3D information under various conditions, and the effect of representational style on the specificity with which a model is interpreted.

Broader Impacts: This research will provide a basis for encouraging and deepening the interdisciplinary collaboration between faculty in computer science (specializing in computer graphics and visualization) and faculty in architecture, two fields with vastly different cultures but with many common interests and objectives. The project will attract and support students from non-traditional backgrounds to work and study in this interdisciplinary area. The modeling and rendering software developed as part of this project will enrich undergraduate and graduate courses in architectural design, and will serve as a resource for multiple potential uses by members of the greater design community. Further, the PIs and their research assistants will contribute to the creation of a virtual environment infrastructure that, through their involvement with the St. Paul Consortium, will be made accessible to city planners and community advocates for use in making informed decisions about the impact of proposed major development projects on the urban landscape.

Abstract
Information Technology Research (ITR) Medium Proposals


Proposal Number: CTS-0324898
Principal Investigator: Ivan Marusic, Victoria Interrante, and Ellen Longmire
University/Institution: University of Minnesota

ITR: Dynamic methods for Identifying, Visualizing and Tracking Eddy Evolution in Experimental Turbulent Flows
This research program unites data visualization and experimental fluid mechanics specialists to investigate and understand the dynamic evolution of key physical mechanisms in turbulent flows. To accomplish this, the evolving nature of eddies, or vortex structures, fundamental building blocks in wall-bounded turbulent flows, will be examined. Experiments will track, in real time, vortex packets, which are structures that have been identified as a key mechanism in producing skin-friction drag in these flows. A dual stereo-PIV system will be traversed along the length of a water channel flow while implementing a feature-identification scheme based on visualization tools to generate a feedback signal. The experiments and feature detection strategies will provide answers to unique and challenging questions about the generation, development, merging and interaction, and breakdown of vortex packet structures and other dominant flow features in wall turbulence. Multivariate visualization methods appropriate for the novel experimental data will be developed. The methods will be applied to see what structures exist, and how they develop and decay. Because vortex packets are characterized by several parameters, the challenge is to present the visualization in a meaningful and useful way to turbulence practitioners. The multivariate visualizations will be optimized with special attention paid to quantifying how and why different visual features, such as color, texture, topography, and motion, work to convey information efficiently and effectively.
This research will provide, direct insight into the dynamic evolution processes of key physical mechanisms in wall turbulence at moderate to high Reynolds numbers. The new insight will constitute a major advance in the fundamental understanding of turbulent boundary layer flows and can lead to new drag reduction strategies and new turbulence simulation schemes. Also, visualization methods suited to complex turbulent flows will be developed that will prove useful to the broader turbulence community as well in researchers in additional fields. This research will lead to unique movies of vortex evolution that will be developed into an educational resource for graduate programs and broader scientific audiences. In addition to graduate students, undergraduates and high school physics teachers will participate in short-term investigations during the summers. The teachers are expected to bring their experiences with cutting edge measurement techniques and engineering applications back to the classroom.

In this project, the PI will lead an interdisciplinary team in a quest for fundamental insights into how to effectively harness the full potential of virtual reality technology for visualization and design. The specific focus of the research will be on architectural applications. Within this context, the PI will seek to enable accurate and intuitive spatial understanding of large, immersive 3D virtual environments presented both via head mounted displays and on large projection screens. For environments presented via a head mounted display, the PI will explore how the accuracy of the user's spatial perception and sense of presence is affected by factors such as: providing a faithful representation of the user's body; providing low latency visual and haptic feedback about the sizes and distances from the user of tracked objects that co-exist in both the real and virtual environments; and providing spatialized 3D ambient sound cues. For information presented via a stereoscopic large screen rear-projection system, the PI will explore how the viewer's ability to attain a maximally accurate intuitive spatial understanding of an interior or exterior is impacted by questions such as: the importance of presenting people with an image of a scene that is generated from a viewpoint that is as close as possible to their own eye position, both laterally and in distance from the ground; the conditions under which the viewer is likely to adopt an interpretation of size and distance relationships in a virtual environment shown via a projection system that is based on the assumptions that underlie interpretations of size and distance in pictures, as opposed to interpretations of size and distance in directly viewed scenes; and whether, when considering display on a large screen, bigger is always better, or if the maximum benefit comes from displaying a scene at "life size" with the possibility that negative consequences might arise from displaying things too large. The PI will further explore the design and evaluation of improved metaphors for enabling intuitive locomotion through very large scale immersive virtual environments, as well as the effective use of abstraction for the representation of uncertain or ambiguous information in such environments.

Broader Impacts: This work will lead to basic observations and rules of thumb derived from careful human subjects experiments that will inform a broad range of research efforts involving the use of virtual environments, in such domains as scientific and information visualization, situational awareness, and diversity training. It will also result in improvements in architectural education stemming from the effective use of virtual environment technology to teach fundamental concepts in visual imagination and integration of an egocentric perspective into the earliest stages of the design process.

Terra Populus: A Global Population/Environment Data Network (TerraPop) will develop organizational and technical infrastructure that will integrate, preserve, and disseminate data describing changes in the human population and environment over time. A plethora of high-quality environmental and population datasets are available, but they are widely dispersed, have incompatible or inadequate metadata, and have incompatible geographic identifiers. The project will enable researchers to identify and merge data from heterogeneous sources to study the relationships between human behavior and the natural world.

TerraPop will focus on four specific kinds of data: (1) census and survey microdata describing the characteristics of individuals and their families and households; (2) aggregate census and survey data, describing the characteristics of places, including aggregate population characteristics, land use, and land cover; (3) remote-sensing data describing land cover and other environmental characteristics; and (4) climate data describing temperature, precipitation, and other climate-related variables. All four data types have an important temporal dimension; most of the data span the past five decades, and some sources reach back to the nineteenth century. TerraPop will make these data interoperable across time and space, disseminate them to the public and to multiple research communities, and preserve them for future generations.

Understanding of interactions between population and the environment has been hampered by the dearth of internationally comparable data. This infrastructure will expand the quality and quantity of such data while making them highly interoperable and easily accessible. Population data closely integrated with data on the environment will allow us to describe the unfolding transformation of human and ecological systems. Data on the human population are crucial for understanding changes in the Earth?s biological and climate processes; equally important, data on climate and land provide essential tools for understanding the impact of environmental change on human behavior. By creating a framework for locating, analyzing, and visualizing the world's population and environment in time and space, TerraPop will provide unprecedented opportunities for investigating the agents of change, assessing their implications for human society and the environment, and developing policies to meet future challenges. The data collection and its analysis tools will contribute to education and public understanding. It will allow teachers to integrate research and teaching, bringing the excitement of discovery into the classroom from primary school to graduate school. More broadly, TerraPop will be a model for the sustainable expansion, maintenance, and improvement of a global data resource.

Immersive virtual environments offer tremendous potential for fundamental and transformative advances in education, training, rehabilitation, architectural design, as well as a wide range of other application areas. The technology has special potential to enhance the process of architectural design by enabling designers to work with their ideas at full scale from the earliest stages of the design process, and to experience the interior spaces of their designed structures from a firsthand egocentric perspective before they are built. The research and educational opportunities made possible by the requested equipment will enable designers and their clients to experience a virtual environment as if the designers and their clients were truly standing in the physical environment that is represented by the virtual environment, and to make decisions based on their experience in the virtual world that are equivalent to the decisions that they would have made as a result of a similar experience in the physical world. The project will teach students of architectural design the value of developing their design ideas from an egocentric as well as allocentric perspective. Through this work, architects, and their clients, will be able to make reliable design decisions that can enhance the livability of a planned space based on their first-person experience of the 3D spatial layout. The project will enable archaeologists, historians, city planners, tourists, and others to gain an intrinsic, egocentric understanding of the spatial layout of large, remote sites through virtual exploration.

Intellectual Merit

An interdisciplinary team consisting of an associate professor of computer science in the college of science and engineering and an associate professor of architecture in the college of design requested funding for three pieces of major equipment and related supplies that will enable them to (a) pursue a multi-faceted research agenda in the development of groundbreaking methods to enhance the cognitive and perceptual realism of immersive virtual experiences, leveraging fundamental insights from visual perception and cognition, and (b) maximize the potential of virtual reality technology to fundamentally enhance the process of design education, emphasizing the importance of integrating an experiential understanding of planned spaces into the earliest stages of the design process. The proposal requests funding for the purchase of (a) a wide field of view head mounted display with an embedded stereo eye tracking system that the researchers will augment with lightweight, close-range depth sensors and a custom-built dual camera and mirror system to achieve a convergence-adaptable stereoscopic video-see-through augmented reality capability, (b) a set of cameras that will enable real time tracking throughout the full extent of the space available in a virtual reality design lab, and (c) an untethered, moderate field of view head-mounted display with optional optical see through capabilities that will, in conjunction with a backpack-worn laptop computer, allow unencumbered free physical movement through large virtual spaces and, in conjunction with the other head-mounted display, enable the project to pursue research in self-embodied multi-person interaction in immersive virtual environments.

Broader Impact

The requested equipment will benefit society by permitting research that will advance an understanding of how people can have experiences in immersive virtual environments that are equivalent to experiences in real world environments. This equipment will support significant advances in design education by enabling the broader effective use of virtual environments technology in teaching fundamental concepts of visual imagination and will support closer interdisciplinary collaboration between faculty and students in computer science, architecture and design. The project will promote science and engineering to the general public through community outreach such as through frequent lab tours for local K-12 students, and will promote participation in science through mentoring and summer programs for middle and high school students.

This is funding to support participation by approximately 12 graduate students (9 domestic and 3 international), along with 5 senior members of the community (faculty and industry researchers), in a Doctoral Consortium (workshop) to be held in conjunction with IEEE Virtual Reality 2014 conference that will take place March 29-April 2 in Minneapolis, and which is collocated with the IEEE Symposium on 3D User Interface. Virtual reality (VR) is a multidisciplinary field involving human-centered computer simulations that seek to imitate or augment real world senses (usually sight, sound, and touch) and experiences. VR research includes developing and assessing methods and systems, and facilitating and understanding user perceptions, beliefs, and behaviors. First organized by the IEEE Computer Society in 1993 and held annually since 1995, IEEE Virtual Reality is the premier international conference and exhibition in this field and includes technical paper presentations, workshops, tutorials, research demonstrations, and exhibits from industry. The IEEE Symposium on 3D User Interfaces has been held annually and colocated with IEEE VR since 2006. Based on prior year numbers, the combined 2014 conferences are expected to attract 350-400 registrants, 15-20 exhibitors, and 30 student volunteers. More information may be found online at http://www.ieeevr.org/2014.

This year's IEEE VR Doctoral Consortium is inspired by and similar in format to last year's successful inaugural event, which was also supported by NSF. The main activities of the DC will take place on Sunday, March 30, immediately prior to the start of the IEEE VR conference proper and in parallel with the second day of the IEEE Symposium on 3D User Interfaces; additional smaller activities will take place at scattered times over the rest of the week. The workshop will include morning and afternoon sessions in which each student presents his/her work to the other student participants and a panel of senior VR researchers, with sufficient time set aside after each talk for discussion and constructive feedback that addresses the strengths of the work, challenges and issues that may arise, and implications of the results. A group working lunch attended by all the students and mentors will be particularly valuable for unifying the individual goals and projects presented within the group as part of a "big picture" envisioning of the future of the field over the next 10-20 years. Each student will also be asked to prepare a poster on his/her research that will be on display during the conference as part of the poster sessions, and to submit a short (2-page) abstract to be archived in the IEEE Digital Library.

Broader Impacts: The goal of the Doctoral Consortium is to provide an interactive and supportive mentoring opportunity for mid-level graduate students in virtual reality, to afford these students a valuable opportunity to get independent perspectives on their research from senior individuals with a wide collective breadth and depth of knowledge, and to build a cohort of young researchers within the VR community. The organizers will make a special effort to recruit participants from groups traditionally underrepresented in computer science, including women, persons of color, and persons with disabilities. They will take steps to achieve diversity among the students with respect to research topics, disciplinary backgrounds, methodological approaches, and home institutions (for example, no more than one graduate student will be accepted from any given institution).

Immersive virtual reality (IVR) technology has the potential to fundamentally transform the architectural design and building industries, by enabling individuals and groups to achieve an accurate and intuitive experiential understanding of a designed environment before it is built. To help realize this promise, the PI and her team will in this project pursue a multi-faceted research agenda that seeks to develop and deploy low-cost, multi-user, head-mounted-display (HMD) based IVR technology to more effectively support the decision-making process in architecture / engineering / construction design reviews. The research will address two critical challenges: to enable individual stakeholders to achieve a more accurate understanding of the 3D spatial structure and affordances for action in a designed interior space and to reliably assess its aesthetic and functional suitability under realistic use conditions; and to facilitate effective collaborative design review by enabling groups of stakeholders to be jointly immersed in the virtual model during the review process. Project outcomes promise to offer immediate benefits to architects, builders, and their clients, who will be able to more readily evaluate the suitability of designed spaces to meet their needs. The work also has the potential to enhance the effectiveness with which IVR technology can be used for a wider variety of collaborative and experiential purposes, in areas from education and training to psychotherapy and rehabilitation. In addition, the research has the potential to support significant advances in design education, by enabling the broader effective use of IVR technology in teaching fundamental concepts of visual imagination, and to foster closer interdisciplinary collaboration between faculty and students in computer science, architecture and design.

The ambitious research plan consists of three key components. The PI and her team will develop robust methods for providing individual users with the ability to see a visually faithful, dynamically rendered 3D representation of their own body while they are physically moving about within an HMD-based IVR, along with a quantitative assessment of the impact of alternative embodiment methodologies on the accuracy of peoples' spatial perception judgments in the virtual environment. The team will also develop robust methods for providing multiple users with the ability to see each other while being co-located in a wide-area, HMD-based IVR, along with a qualitative assessment of best-practice representational approaches for supporting effective interpersonal communication and simultaneous accurate spatial understanding when multiple users are immersed as a group in a shared virtual environment. Finally, the team will create a system for populating a designed environment with realistically behaving and interactively responsive autonomous virtual agents, along with qualitative and quantitative testing of the impact on spatial perception and functional suitability judgments from experiencing dynamically populated, as opposed to static or unpopulated, immersive virtual environments, plus related efforts that will assess the subjective and objective realism of the agents' dynamic behavior.

The human senses provide the information to the brain about the world around us. Deficits in visual, auditory or other sensory inputs have a major impact on quality of life, including education, employment and social engagement, which in turn places a major burden on the US economy. The scale of this problem is large: According to reports by the World Health Organization (WHO) and the National Institutes of Health, there are approximately 5 million Americans with vision impairment and around 36 million Americans with some form of hearing loss. There is broad recognition of the need for interdisciplinary collaboration for translational research on disabilities. Research into the development of more effective assistive technologies and environmental modifications requires interdisciplinary expertise that unites a fundamental understanding of the basic sensory science (vision, audition, motor control, speech and language) with deep technical expertise in engineering, computer science, and other related fields. This National Science Foundation Research Traineeship (NRT) award to the University of Minnesota will enable the formation of an integrated and interdisciplinary training program in sensory science through the joint strengths of the university's Center for Cognitive Sciences and Center for Applied and Translational Sensory Science. The program's goal is to prepare future scientists to combine engineering approaches with scientific knowledge and methods so that they are in a position to develop the next generation of sensory aids that will improve the quality of life for Americans with sensory loss. The program will serve students from computer science, engineering, kinesiology, psychology, and speech-language-hearing disciplines through courses, research opportunities, internships in the medical-devices industry, and public outreach activities. A total of 49 different PhD students will be enrolled in the program over the course of the 5 years, 18 of whom will receive NRT fellowships.

This project creates a new interdisciplinary graduate training program that encompasses the following key aims: 1. Establish an academic program in the form of a graduate minor through which students will receive the focused, multi-disciplinary educational background they will need to address critical challenges in the development of assistive technologies for sensory loss. 2. Advance interdisciplinary research in translational sensory science through the development of structural mechanisms that support the formation of interdisciplinary research teams. 3. Provide extra-curricular training and networking opportunities to the trainees through supplementary mechanisms, including weekly journal clubs, summer and winter workshops, spring and fall research symposia, and an annual fall retreat. Students will also participate in public engagement such as preparation of podcasts on research topics of interest to the stakeholders. 4. Provide practical career development opportunities through personalized contacts with local and national companies. Many trainees will eventually seek employment in industry, and those who continue in academia should seek to engage in collaborative work with industry partners. In an effort to provide students with real-world experience that can inform their research and post-graduation career choices, the program will offer the trainees 8-10 week summer internship opportunities in industry-based applied research.

The NSF Research Traineeship (NRT) Program is designed to encourage the development and implementation of bold, new potentially transformative models for STEM graduate education training. The Traineeship Track is dedicated to effective training of STEM graduate students in high priority interdisciplinary research areas, through comprehensive traineeship models that are innovative, evidence-based, and aligned with changing workforce and research needs.

With a global installed user base of over 28 million people, virtual reality is a rapidly advancing field with numerous emerging applications in education, training, rehabilitation, healthcare, social communications, and entertainment. However, the effectiveness of virtual reality applications and their rate of public adoption is currently limited by the fact that many users experience physical discomfort during or after their use, with symptomatic characteristics indicative of motion sickness. This problem, known as "simulator sickness" or "cybersickness", is one of the most significant usability challenges for users, developers, and stakeholders of immersive technologies. This project offers a novel and empirically-grounded research methodology to study, predict, detect, and ultimately mitigate simulator sickness, which can substantially improve both the subjective user experience and the effectiveness of current and future virtual reality applications. Furthermore, prior research has shown that motion sickness disproportionately affects women. This project seeks to advance understanding of these differences and develop adaptive strategies for mitigating simulator sickness on an individual level, which can ultimately increase the overall number of potential users worldwide and erode the inequitable barriers that currently exist for engaging with immersive technologies.

This project seeks to address simulator sickness through a systematic effort that will advance fundamental understanding of the relationship between motion kinematics and the adverse symptoms commonly experienced by users of virtual reality systems. Specific activities include the following: (1) development of models that predict the likelihood of experiencing simulator sickness based on an individual's motion characteristics; (2) introduction of real-time methods for early detection of sickness onset before the user experiences discomfort; (3) identification of specific problematic virtual reality stimuli that are associated with simulator sickness; (4) development of adaptive mitigation strategies to reduce the likelihood and severity of adverse symptoms; and (5) rigorous experimental evaluation of the effectiveness and tradeoffs of newly developed techniques. The project offers methodological innovation in several areas related to the fundamental study of motion sickness phenomena, including the investigation of eye gaze stability for the prediction or early detection of simulator sickness. Additionally, a key innovation is that the data collected from empirical studies will be utilized to develop adaptive techniques that adjust automatically based on the individual's predicted sickness levels and current real-time state, both measured through quantitative motion kinematics. The project will also result in the creation of a large-scale motion kinematics dataset that will be made publicly available for future research.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project, developing an instrument to enable a pioneering way to canonically represent and systematically quantify behavioral manifestations for a suite of neuropsychiatric disorders, integrates a powerful computer cluster with various sensor modalities and associated immersive technology components that facilitate data analysis and perception of physical and virtual experiences. The instrument will bring together neuropsychiatrists and engineering scientists seeking the development and deployment of tools that promote discovery and new knowledge in their respective domains, and innovation with immediate societal impact. It aims to enable medical experts to explore clinical hypotheses during diagnosis and treatment. Concurrently, scientists/engineers will explore the implementation of new computational tools and algorithms that satisfy new requirements utilizing a set of clinical hypotheses.

Challenging the current understanding of a spectrum of neuropsychiatric conditions and triggering the explorations of new ways to affect them, this instrument should contribute to our understanding of the presence and evolution of mental states. Its unique characteristics will help in integrating a range of informational cues (e.g., visual, acoustic, haptic) along with virtual simulated information. This interplay between physical and virtual experiences and stimuli and their perception in an immersive environment constitutes the ultimate objective which will unleash creativity in better understanding and treat mental illnesses.

The tight coupling of hardware and software components will enhance the various types of interactions such as expert-patient, and patient-virtual objects. Specifically, the instrument will support the following research efforts:
- Comprehensive compilation of behavioral data,
- Development of visualization tools,
- Delivery of haptic information feedback,
- Creation and interaction of virtual objects, and
- Enhanced sentiment understanding from audio/visual signals, facial expression representation and analysis.
The instrument enables the creation of communication channels with clinics across the country. Moreover, it contributes in forming a strong foundation to explore and understand a broad variety of neuropsychiatric disorders manifested via verbal and non-verbal communication pathways, aims to form a foundation that promotes strong collaboration targeting real-world challenges with immediate societal and economic impact, and offers an environment for discovery via the strong coupling of data visualization, User Interfaces-User eXperience (UI-UX) design, immersive perception, data exploration, and machine learning.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.