Ramesh Raskar, Phd - US grants

This is a collaborative project leveraging expertise of Ashok Veeraraghavan, William Marsh Rice University (IIS-1116718) and Ramesh Raskar, Massachusetts Institute of Technology (IIS-1116452). Imaging and display devices are all around us and are used in a variety of applications. The spatial resolution, depth range, depth resolution, temporal resolution, frame-rate and bandwidth of these devices are usually fixed a priori. When the resolution and other properties of the content being imaged or displayed does not exactly mimic those that were assumed a priori, this leads to inefficiencies (in utilizing available resources) and undesirable artifacts (aliasing, blurring and noise). Since both imaging and display devices are fast becoming multi-purpose, there is a need to develop imaging and display architectures (and algorithms) that are capable of adapting their resolution and bandwidth characteristics to match those of the content.

The goal of this project is to develop imaging and display devices that adapt to scene, motion, geometry, viewer, or illumination conditions. Such adaptive devices lead to performance improvements and novel capabilities hitherto unexplored. This research agenda is organized into four intellectual thrusts: (1) the establishment a theoretical framework for Adaptive Coded Imaging and Displays (AdaCID) that enables efficient exploration of the space of designs (2) the design of adaptive coded imaging systems that adapt to scene geometry, motion, and illumination (3) the design of adaptive and interactive coded 2D/3D displays that adapt in real-time to content, viewer position, and the human visual system enhancing visual appearance and allowing intuitive 3D interaction and (4) the demonstration of coded feedback projector-camera systems enabling rapid acquisition of range and material characteristics.

It is expected that AdaCID will have far-reaching impact to diverse applications spanning consumer imaging and displays, machine vision and automation, scientific/medical imaging and displays and surveillance. Since AdaCID and the broader field of computational imaging and displays is increasingly important, they will be integrated into various courses offered at Rice University and MIT. Broad dissemination of the educational material will be achieved through participation in the free, open-licensed Connexions program and OpenCourseWare and in public-domain museum initiatives (at the MIT Museum). This project also offers collaborative research opportunities for students at the two institutions. Project Website (http://cameraculture.media.mit.edu/AdaCID/) provides additional information.

CGV: Small: Inverse Light Transport under Femto-Photography and Transient Imaging
Raskar, Ramesh, Massachusetts Institute of Technology

How can you photograph objects beyond the line of sight? How can you recover bidirectional reflectance of materials from a single viewpoint? These seemingly impossible tasks are possible by considering the finite speed of light and using a new type of computational photography called, Femto-Photography. New advances in ultra-fast imaging provide tremendous new opportunities in modeling, representing and synthesizing light transport in computer graphics and computer vision. Research in computational photography and scene understanding will benefit by analyzing the transient response of the scene to extremely short duration active illumination. Traditional imaging uses steady-state response where the global illumination has reached an equilibrium state. The investigators are developing a new theoretical framework for transient light transport and are addressing inverse problems using time-resolved imaging. The investigators have recently developed the first physical demonstration of hidden geometry recovery.

The research aims to develop a new branch of computational imaging by developing a mathematical framework for studying higher dimensional light transport that exploits time-resolved imaging. This research brings ultra-fast imaging in the realm of computer graphics/vision and computational photography. The finely sampled time-dimension provides a range of research directions for modeling and measuring geometry and photometry of scenes that were previously considered beyond the reach of traditional machine vision. The techniques for time-resolved imaging exploit multiplexing, sparsity-exploiting reconstructions, state-space formulation, system identification methods and parameterized reflectance models in novel ways. Overall, the research pushes the boundaries of light transport based methods by an extra (time) dimension and hopes to show that forward and inverse problems in 5D light transport can inspire the next generation of imaging hardware and algorithms.

Diffractive masks and algorithms for light field capture
PIs: Ramesh Raskar, MIT Media Lab Alyosha Molnar, Cornell ECE

Advanced imaging and display technology requires integrated, low cost systems able to efficiently capture and characterize light from 3-D scenes. In particular, a 3-D scene can be described by the collection of light rays it generates, called the light field. This research combines concepts from mask-based light-field imaging with angle sensitive pixels (ASPs). While mask-based light-field capture is much better understood mathematically, and masks are cheaper to manufacture and more easily modified on-the-fly, diffractive ASPs provide smaller, denser light field sensors, and provide naturally compressible outputs. This project combines the physics and signal processing of these approaches to enable optical imaging systems that capture more information than normal cameras while reducing the system's complexity. This work broadly impacts diverse applications spanning consumer imaging and displays, machine vision and automation, scientific/medical imaging and displays, robotic surgery, surveillance and remote sensing.

3-D images and video can be captured by measuring the combined spatial and angular distribution of light (the light field). This research combines two techniques for light-field capture: mask-based light-field imaging and diffractive angle sensitive pixels (ASPs). A critical element of this work is the development of a mathematical framework that maps between conventional geometric light fields and the diffractive optics upon which ASPs rely. A second element is constructing hybrid systems based on this mathematics, leveraging diffractive effects in mask design, and combining masks with ASPs in single light-field cameras. This work also combines formalisms in existing light field methods with knowledge about real 3-D scene statistics to develop optimal (in the sense of usability and compressibility) basis sets for sampling and encoding the light-field. All of these aspects combine to reduce the size, cost and complexity of light field cameras, while simultaneously enhancing their capabilities.

The proposed technology is a low-cost portable retinal imaging device to diagnose and monitor forms of retinopathy, early enough for medical intervention to prevent vision loss. Images are captured and visualized using computational photography techniques and transmitted to a cloud-based server that performs analysis and allows access to remote experts for evaluation. This approach is not only intended as a substitute for present day cameras that retail for $35,000, but also has the potential to changes the way retinal imaging is performed and can achieve this detection accuracy with minimal prior training of its software interface. Two novel strategies underlie the technology. The first is a technique to utilize indirect diffused illumination of the retina through the sclera instead of illumination with a focused beam through the pupil, thus eliminating the need for a specialist to dilate the pupil and precisely focus the beam. The beam is also used to illuminate different points on the retina. A stimulus presented on a display in front of the opposing eye, not being screened is used to vary the patient?s focus across his visual field to generate the full retinal image. This second novel strategy exploits binocular coupling - the phenomenon of both eyes moving together.

With soaring demand for health care in the coming decades, the proposed approach will provide a cost-effective and simple way to monitor retinal health and can potentially transform the $3 Billion estimated market for retinal imaging. The advantage in price, portability, and simplicity dramatically increases accessibility to retinal screening. The application can easily be expanded to detect and monitor many other retina related disorders, such as glaucoma and age related macular degeneration, as well as other diseases that manifest in the retina. Recent advances in mobile computing, portable imaging, and wireless communication have the potential to enable retinal imaging outside of clinical settings, which could have significant impact on tele-medicine through mobile, non-invasive health screenings. Future possibilities for frequent and out-of-clinic retinal imaging on a large scale may provide new opportunities for the detection, diagnosis and treatment of such diseases.

This research examines the technology challenges with providing services to pop-up cities. Pop-up cities are locations where the population expands very rapidly by an order of magnitude or more in a very short time. One of these examples is provided by the Kumbh Mela in India. This event is expected to attract 30 million pilgrims to the city of Nashik, India in late summer and early fall 2015. This proposal is a RAPID and is designed to collect data on crowd and service needs for a pop-up city. The information will relate to crowd dynamics and infrastructure needs.

The proposal offers the opportunity to collect huge amounts of data that can be used for analyzing infrastructure and services needs due to emergent pop-up cities. The data is time perishable and can only be collected when the event is in progress. The data can be mined to enable the understanding of requirements for smart and connected communities when a rapid community growth occurs over a short but extended period. The research will provide insights into how much infrastructure needs to be built-up well in advance (for example supporting Olympic Events or similar large gatherings) and how much can be created just in time. It will help identify service needs spanning many of the technologies needed by smart and connected communities.

This project fundamentally combines the emerging time of flight imaging techniques with computational methods to redefine a camera and also go beyond the conventional barriers in scientific imaging. Imaging has transformed science and technology in many fields. Time-aware ultrafast imaging can bring further radical new innovations in coming years. Recently, there has been a significant commercial interest in converting time-aware sensors into low cost consumer solutions. Going forward, solving time-based forward and inverse transport problems can impact new fundamental research in biology, physics, optics, computer science, engineering, and mathematics, with broad applications in health, robotics, defense, and mobility. They have high potential to stimulate economic investment and entrepreneurship using modern imaging solutions.

Emerging image sensors with picosecond (ps) time resolution provide new ways to capture and understand the world. For scene analysis, typical computational imaging techniques exploit sensor parameters such as spatial resolution, wavelength, and polarization. However, they are far slower than light speed and are consequently limited in their ability to model the complex dynamics of light propagation. Time-resolved (or transient) sensors overcome this limitation, but their integration with computational methods has not been realized yet. Therefore, with the recent spurt in commercial time-of-flight (ToF) systems, new research in transient computational imaging is well-timed. Beyond ToF depth information, this research explores the capture and analysis of per-pixel time profiles at ps scales. This leads to joint re-examination of fundamental inverse problems and solutions in scientific, industrial and consumer applications. Specifically the project builds computer vision algorithms for seeing objects beyond the line of sight, behind diffusive layers and inside turbid media. This provides novel applications in medical imaging. With the development of the theoretical foundation and enabling tools, the project accelerates research and commercialization of this new field.

The objective of this project is to develop and deploy a privacy-first digital solution for public health coordination including contact-tracing to curb pandemics like COVID-19 spread. The key is to provide location and context for citizens and public health experts. Current approaches operate on a trade-off between privacy and effectiveness, relying on general public broadcasting that introduces uncertainty in the information extracted or resorting to privacy-violating technologies that risk individual rights against stigmatization and surveillance. This project will break past this dichotomy by developing a technology-based solution for coordinating information on infection and possible transmission through contact-tracing while protecting the privacy rights of viral carriers and unexposed citizens.

Beyond assisting the containment of COVID-19 pandemic by contact tracing, this project will make empirical contributions to the fields of computing, healthcare, crisis response, and more. With privacy preservation being the key aspect of this project, contact tracing is achieved by using encrypted GPS trails and rotating Bluetooth identifiers. In this approach, redacted information of an infected individual is only shared while no information leaves the device of a healthy person. Specifically, this project will advance knowledge regarding: 1.) how cryptographic techniques can be implemented on ubiquitous platforms like smart phones through easy to use apps to efficiently use privatized data without leakage of any sensitive information; 2) how personal-technology solutions to societal crises can effectively influence behavior and consequently affect the outcome of such crises; and 3) how ?split-learning?, a resource efficient distributed AI technique can be implemented with personal information on health, demographic, travel history, spatial context, and real-world engagement to perform private risk-assessment post contact-tracing to reduce false alarm rates. The solution is being built by a consortium of epidemiologists, engineers, data scientists, digital privacy evangelists, professors and researchers from reputable institutions. This is crucial to reduce disruption in socio-economic activity and keep panic under rationally controllable levels in response to future emergencies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Recent events have shown the value of better data sharing; the rapidity of change in COVID-19 and the difficulty of coordinating response is a clear example. At the same time, public concern over privacy is increasing, making it more difficult to collect reliable data. Improved use of government data by a wide range of users has potential to improve research as well as day-to-day operations at a variety of levels, from local to national and global. Government data can also foster innovation, providing opportunities for small companies based on large-scale data analysis that are increasingly reserved for large organizations capable of gathering their own data (often in ways that raise privacy concerns). We are running a 2-day workshop to identify challenges and explore mechanisms (technical, legal, and procedural) to enable greater use of privacy-sensitive data held by government agencies.

Privacy technology has experienced great advances over the past decade. Breakthroughs such as fully homomorphic encryption, differential privacy, and advances in secure multiparty computation have great promise to enable broader public use of data, while maintaining individual privacy. While there have been some successes, there are many applications where there is a significant gap between what technology offers, data needs of users, and policies and procedures to ensure privacy. This includes best practices as well as research and policy challenges that limit effective use of data. This workshop explores these concerns, highlighting new research challenges based on actual user needs for government data. This covers research in underlying technologies that address real-world challenges, as well as research in policy to enable use of technology in ways that provide appropriate levels of protection for privacy-sensitive data. The outcome will be a report identifying unmet needs of data users, issues limiting broader use of government data, and technologies and policies that show promise of bridging these gaps. This will serve as a roadmap for research investment to address these challenges.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The objective of this project is to develop and deploy a privacy-protecting, user-centric, digital solution to enhance vaccination coordination and to create a privacy-preserving, data-aggregation platform for researchers. The project will enable a decentralized, end-to-end protocol that spans the entire vaccine user journey, from enrollment in phased vaccination to long term monitoring of adverse effects. The team will consider health equity from the perspective of trust, privacy, and inclusivity. Current systems for vaccination coordination focus on a single part of the system or require a smartphone for every vaccine recipient which aggravates the equity concerns. This project addresses such concerns through an array of user-facing solutions: QR codes on paper vaccination cards which can operate offline as well as mobile phone apps without live internet access. The standardized data sharing system consolidates both population-wide and individualized information in a single platform to increase the speed and effectiveness of the intervention, vaccination in this case, so that it can be monitored and analyzed. This enables a bird’s eye view of the cyber-physical-social ecosystem without creating a surveillance state. The project will push the boundaries of data-driven predictive analytics for pandemic response and pandemic preparedness.


The project’s goal is to provide a user-centric solution that can aid researchers and planners of the current and future pandemics. In this project, the user’s journey and the relevant de-identified data collection is divided into four parts: (i) Digitally enhanced enrollment system for phased vaccination using digitally signed coupons, (ii) A privacy-preserving QR code based vaccination card, and a smartphone app to interface with vaccination sites without revealing any personally identifiable information to centralized servers, (iii) Proof of vaccination in a tamper-evident and secure manner available with digitally signed offline credentials, (iv) Monitoring and alert systems for adverse reactions that enable users to upload their symptoms in a cryptographically authenticated manner. The project involves building data aggregation and data dissemination solutions with varying levels of granularity for population-scale and individual scale analysis. For aggregation, to preserve the privacy of early contributors, the project will use a new generation of techniques based on secure multi-party computation. For dissemination, the project will use Split Learning and Split Inference methods invented by the investigator at MIT that may be able to better address privacy-utility trade-offs. Vaccination data is critical at all three levels: (i) logistics and monitoring of vaccines (ii) vaccination workflows and (iii) user experience before and after vaccination. This project will generate tools for an efficient data gathering monitoring system for future pandemics and emergencies of a similar nature without invasion of personal freedoms. The systems and methods are being built by a consortium of epidemiologists, engineers, data scientists, digital privacy evangelists, professors, and researchers from various institutions. Such a diverse collaboration is essential to minimize disruption to the existing vaccination system and ensure a smooth vaccination roll-out across the nation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The COVID-19 pandemic has highlighted human vulnerability to infectious disease outbreaks and reinforced the need for improved data-driven response and preparedness. The BEHIVE (BEHavioral Interaction and Viral Evolution) research team for Predictive Intelligence for Pandemic Prevention (PIPP) aims to tackle a fundamental challenge in disease outbreak prevention by integrating the study of human behavior using a computational data-driven lens. The impact of human behavior and social interactions remain underutilized in efforts spanning scenario development, forecasting, and epidemic mitigation. Proposed educational and outreach activities will train early career researchers and practitioners in PIPP. The trans-disciplinary nature of the project including connections with the humanities will encourage diverse cohorts of researchers to engage in this public-facing field.<br/><br/>Several accelerating trends, such as widening data collection, new Artificial Intelligence (AI) and Machine Learning (ML) techniques, high fidelity computational modeling and recent surge in social/behavioral knowledge due to COVID-19, create an opportunity to tackle the challenge of integrating human behavior into epidemic response using a synergistic team-science approach. Project will develop methods incorporating behavioral feedback to bridge mechanistic and AI models, to provide specificity and context needed via genomic surveillance and robust predictions, and to empower coordinated decision making by building strategic portfolios. These efforts will lead to novel AI/ML frameworks, new modeling/decision-making paradigms, facilitate early warning systems and inform mitigation efforts to prevent pandemics and reduce risk of outbreaks in the first place. The diverse interdisciplinary team consists of computer scientists, biologists, engineers, and behavioral scientists, along with experts in medicine and public health (from Georgia Tech, MIT, Michigan, UGA and Mayo Clinic) with broad and significant expertise in epidemic research spanning foundational research and intervention-focused work during the COVID-19 pandemic. The project team will also partner with multiple academic, industrial, government public health and non-profit setups, which will help enlarge the impact of the proposed research. <br/><br/>This award is supported by the cross-directorate Predictive Intelligence for Pandemic Prevention Phase I (PIPP) program, which is jointly funded by the Directorates for Biological Sciences (BIO); Computer, Information Science and Engineering (CISE); Engineering (ENG) and Social, Behavioral and Economic Sciences (SBE).<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.