Joshua Tenenbaum - US grants

To model an individual's choices under uncertainty, theorists typically assume the choices made maximize the individual's utility. While frequently a good description of observed behavior, there are instances where people instead choose alternatives in proportion to their associated probabilities of reward. This probability matching behavior is sub-optimal. Probability matching behavior and optimal behavior would both result depending on the time available to make decisions (where more time produces more optimal decisions) if individuals base their choices on a sampling algorithm. In this Doctoral Dissertation Improvement grant, the PI will test whether such an algorithm is responsible for observed choices and, furthermore, whether people are optimally suboptimal (i.e., optimal in their decision regarding when to be more, or less, optimal.

To test these hypotheses, experimental subjects will be assessed in terms of how flexible they are at making tradeoffs between speed and accuracy in motor decisions under uncertainty and how generic decision processes are across decision domains. Subjects are then tested for whether their decisions under cognitive stress deteriorate to probability-matching, as predicted by the proposed algorithm. Finally, subjects will be tested using fMRI to determine whether one brain structure represents expected utility arising from different sources of uncertainty. This research holds promise for reconciling models of humans as ideal agents with established failures and limitations of human decision-making.

In order for robots to collaborate with humans, they need to be able to accurately forecast human intent and action. People act with purpose: that is, they make sequences of decisions to achieve long-term objectives. For instance, in driving from home to a store, people carefully plan a sequence of roads that will get them there efficiently. In predicting a person's next decision, algorithms must be developed that reflect these purposeful actions.

Currently, robots are unable to anticipate human needs and goals, and this represents a fundamental barrier to their large-scale deployment in the home and workplace. The aim of this project is to develop a new science of purposeful prediction that can be applied to human-robot interaction across a wide variety of domains. The work draws on recent techniques based on Inverse Optimal Control and Inverse Equilibria Theory that enable statistically sound reasoning about observed deliberate behavior. These new methods provide the foundations of a theoretical framework that integrates traditional decision making techniques like optimal control, search and planning with probabilistic methods that reason about uncertainty and hidden information, particularly about goals, utility and intent.

Intellectual merit: The project will provide a general framework that allows robots to anticipate and adapt to the activities of their human co-workers based on perceptual cues. The investigators will develop the theory, a computational toolbox, and, in collaboration with industrial partners, prototype deployments of these new methods for the prediction of peoples' behavior in a diverse set of robotics domains from computer vision to motor control. The project is transformative in that it combines a novel theoretical/algorithmic framework with extensive support in terms of volume of data and validation infrastructure in the context of many applications.

Broader impacts: A revolution in personal robotics in both the home and workplace depends on the ability to forecast human activities and intents; small- and medium- scale manufacturing will make a leap forward through agile robotic systems intelligent enough to understand and assist their co-workers in flexible assembly tasks; and robust models of pedestrian and vehicular traffic flow will enable more effective driver warning systems and safer autonomous mobile robots. Purposeful prediction technology is an important step towards enabling such understanding of actions and intents in these arenas. The research work will involve the training and mentoring of undergraduate, masters and doctoral students as well as post-doctoral fellows in this emerging multi-disciplinary research area at the intersection of computer and cognitive sciences and robotics.

People are able to reason about the world in amazingly complex ways, yet we consider these capacities part of simple “common sense,” generally shared across individuals and cultures. We toss and catch balls, stack dishes in the sink, and pour a morning cup of coffee with almost no effort. Yet the cognitive systems that support these capabilities are not well understood; even our most advanced attempts to reverse engineer them in robots fall short of human-level efficiency or flexibility. This grant was designed as an “adversarial collaboration” to bring together scientists from two different sides of a critical debate about the nature of human physical reasoning abilities. One theory (championed by the MIT PIs) suggests that this physical reasoning is based on a cognitive system that allows people to simulate what might happen next, similar to how physics engines for video games are used to predict what will happen next in those scenes. While this theory has provided many successful explanations of human behavior, including making precise predictions about how people think Jenga towers will fall, or where they think balls flying through the air will land, another growing body of research (led by the NYU PIs) has demonstrated many instances where the simulation theory cannot adequately describe what people do, but where simpler and approximate “rules-of-thumb” (even inaccurate ones) can. Because human physical reasoning is unlikely to be purely simulation or purely based on simplified rules, a team of experts from both sides of this debate will be crucial for advancing our understanding of the cognitive processes that underlie these reasoning capabilities. Towards reconciling these views, this grant advances the idea that consideration of known human limitations -- e.g., in memory or attention -- can explain the processes that people use when reasoning about the physical world. The goal is to integrate these constraints into a more complete theory of human reasoning that can account for both our failures and our successes in comprehending the physical world. True understanding of these processes will require “reverse engineering” human cognition and perception by designing computational models with similar limitations and capabilities to people. These scientific models may provide insight for researchers in AI and robotics who are interested in designing systems that interact with the world like people, including self-driving cars or the control of prosthetic limbs. Furthermore, exploring how people learn and reason about physics may provide new approaches for physics education. Finally, studying and modeling these facets of physical reasoning will require developing extensible tools, which will be released as open-source software to open up the research into human physical reasoning to a wider set of scientists.

This project studies and proposes to resolve tensions between theories of human physical reasoning that suggest that it is based on relatively accurate simulatable mental models, and those that suggest it is based on heuristics and other qualitative forms of reasoning. The research includes experiments related to those that have been used to demonstrate simulation theory, but modified to induce shortcuts in physical reasoning in two broadly different ways. Aim 1 experiments consider scenarios that are expected to run into human resource limitations, either in attention, memory, or time – for instance, asking people to predict the stability of complex towers of blocks with too many pieces to track individually. Aim 2 experiments consider scenarios that could be reasoned about with simulation, but could more easily be reasoned about with simple rules or heuristics – for instance, studying how people use rules like “the heavier side will tip over” when judging which direction a balance beam stacked with objects will fall. Human behavior in these experiments is examined for deviations from pure simulation theory in line with the expected resource limitations (e.g., using rules, focusing on a subset of objects, or representing objects more coarsely), and computational models are developed to explain this behavior. These models are designed around the framework of “resource-rational” cognition, which suggests that people deploy limited cognitive resources in a way that efficiently solves the problems they encounter. The behavioral results and models together allow investigation into (a) whether and when people’s physical reasoning is constrained by resource limitations, and (b) the types of shortcuts people take to circumvent these limitations. Performing this research requires developing an integrated software suite for designing experiments and modeling across a wide variety of physical scenarios. Designing these integrated packages typically requires a large set of technologies -- physics simulators, graphics engines, computational modeling methods -- that are outside the reach of most psychologists, which in turn limits research into human physical reasoning. The PIs are in a unique position to contribute here because their laboratories are focused on computational models of psychology and they have an extensive track record of developing open-source software used by multiple research groups worldwide. The software suite used is designed to be open-sourced and shared with the broader research community to facilitate further research into human physical reasoning without requiring extensive knowledge of the underlying technologies.

This work was supported by SBE/BCS Perception, Action, and Cognition, EHR Core Research (ECR), and CISE/IIS Robust Intelligence.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The last ten years have witnessed an astonishing revolution in AI, with deep neural networks suddenly approaching human-level performance on problems like recognizing objects in an image and words in an audio recording. But impressive as these feats are, they fall far short of human-like intelligence. The critical gap between current AI and human intelligence is that, beyond just classifying patterns of input, humans build mental models of the world. This project begins with the problem of physical scene understanding: how one extracts not just the identities and locations of objects in the visual world, but also the physical properties of those objects, their positions and velocities, their relationships to each other, the forces acting upon them, and the effects of forces that could be exerted on them. It is hypothesized that humans represent this information in a structured mental model of the physical world, and use that model to predict what will happen next, much as the physics engine in a video game generates physically plausible future states of virtual worlds. To test this idea, computational models of physical scene understanding will be built and tested for their ability to predict future states of the physical world in a variety of scenarios. Performance of these models will then be compared to humans and to more traditional deep network models, both in terms of their accuracy on each task, and their patterns of errors. Computational models that incorporate structured representations of the physical world will then be tested against standard convolutional neural networks in their ability to explain neural responses of the human brain (using fMRI) and the monkey brain (using direct neural recording). These computational models will provide the first explicit theories of how physical scene understanding might work in the human brain, at the same time advancing the ability of AI systems to solve the same problems. Because the ability to understand and predict the physical world is essential for planning any action, this work is expected to help advance many technologies that require such planning, from robotics to self-driving cars to brain-machine interfaces. Each of the participating labs will also expand their established track records of recruiting, training, and mentoring women and under-represented minorities at the undergraduate, graduate, and postdoctoral levels. Finally, the collaborating laboratories will continue and increase their involvement in the dissemination of science to the general public, via public talks, web sites, and outreach activities.

Deep neural networks have revolutionized object recognition in computers as well as understanding of object recognition in the primate brain, but object recognition is just one aspect of vision, and the ventral stream is just one of many brain systems. Studying physical scene understanding is a step toward scaling this reverse-engineering approach up to the rest of the mind and brain. Predicting what will happen next and planning effective action requires understanding the physical basis and physical relationships in the visual world. Yet it is unknown how humans do this or how machines could. Both challenges are addressed in this project by the building of image computable, neurally mappable computational models of physical scene understanding and prediction (Thread I), and using these models as explicit hypotheses for how the brain might accomplish these tasks, which will then be tested with behavioral and neural data from humans (Thread II) and non-human primates (Thread III). This project aims to make a transformative leap in understanding: from small-scale, special-case models and isolated experimental tests to an integrated large-scale, general-purpose model of a major swathe of the primate brain, that functionally explains much of the immediate content of our perceptual experience in every scene that confronts us. The work will advance theory by developing the first image-computable models capable of human-level physical scene understanding and prediction. Beyond understanding of the mind and brain, this research is directly relevant to AI and robotics (which require physical scene understanding), and brain-machine interfaces (which require understanding of the relevant neural codes). For the broader research community, the project will a) develop public datasets, benchmark tasks, and challenges, b) host adversarial collaborations to address these challenges, and c) host interdisciplinary workshops linking research communities from psychology to AI to neuroscience to address the fundamental questions that span these fields.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.