Mark Steyvers - US grants

DESCRIPTION (provided by applicant): Numerous clinical trials have shown that Motivational Interviewing (MI;Miller &Rollnick, 2002) is an efficacious treatment for alcohol use disorders (AUD) and related health behavior problems (e.g., Burke, Dunn, Atkins, &Phelps, 2005), but much less is known about the therapy mechanisms of MI (Huebner &Tonigan, 2007). Process research has typically relied on behavioral coding schemes such as the Motivational Interviewing Skills Code (MISC;Miller, Moyers, Ernst, &Amrhein, 2008). Although MI mechanism research with the MISC has produced some of the best data to date (e.g., Moyers et al., 2007), behavioral coding has a number of limitations: 1) it is phenomenally labor intensive, 2) objectivity, reliability, and transportability of coding can be challenging, and 3) it is inflexible (i.e., any new codes require completely new coding). The current proposal brings together a highly interdisciplinary team to develop linguistic processing tools to automate the coding of the MISC and Motivational Interviewing Treatment Integrity (MITI;Moyers, Martin, Manuel, Miller, &Ernst, 2007). The coding of both systems is based on two types of linguistic data: what is said, and how it is said. Our team members in computer science, cognitive science, and electrical engineering are leading researchers in text-mining and speech signal processing, and their methods will be applied to MI transcripts and recordings to automate coding of the MISC/MITI. The core, methodological tool will be topic models (Steyvers &Griffiths, 2007), Bayesian models of semantic knowledge representation. Topic models identify groupings of words that constitute meaning units (or topics), and a recent extension models coded data (e.g., MISC) in which the model learns what specific text is associated with specific tags. Two specific aims encompass the current proposal: 1) Assess the accuracy of topic models to automatically code the MISC/MITI using transcripts and audiofiles of MI sessions, and 2) Test MI theory (within session and long-term outcome) using approximately 1,167 sessions of MI coded in Aim 1. These aims will be accomplished using three MI intervention studies: two studies focused on college student drinking and one hospital-based study of drug abuse. The long-term objectives are to use innovative linguistic tools to study therapy mechanisms and develop more efficient systems for collecting psychotherapy process data. Alcohol use disorders continue to represent an incredible societal burden in terms of death, health complications, fractured relationships, and economic costs. The current research will provide innovative tools for studying why therapy works, which in turn can help to ameliorate some of the deleterious effects of AUD. PUBLIC HEALTH RELEVANCE: Research focused on psychotherapy mechanisms of alcohol use disorders (AUD) have often relied upon behavioral observation coding schemes, such as the Motivational Interview Skills Code (MISC), which are time consuming and can present difficulties with reliability. The current, interdisciplinary proposal develops methods for automating behavioral coding through applying recent advances in text-mining and speech signal processing.

Understanding the complex determinants of individual health and wellbeing is critical for the promotion and maintenance of a healthy world population. Wellbeing may be understood not only as the absence of physical and mental illness but also as the quality of life and optimal functioning of individuals. It is well known that individuals vary tremendously in terms of cognitive abilities and dispositions, as seen from performance on high-order cognitive tasks, decision-making preferences, and emotional competencies. However, the neural underpinnings of much of this variability are poorly understood: It is unclear how individual differences in brain structure and function across tasks and processes are linked to abilities and competencies. This project explores a mathematical and computational framework for investigating a large-sample neuroimaging and behavioral dataset in order to improve our understanding of individual differences in cognitive performance. An ultimate goal of the project is to predict individual cognitive performance in novel, real-world situations based on observed (past) behavioral and neuroimaging data and contribute to the understanding of cognitive health and wellbeing of individuals. The project will also offer many training opportunities for the next generation of scientists.

The technical approach will build on and integrate recent advances in cognitive science, neuroscience, statistics, and machine learning. Statistical models will integrate data from both brain imaging and behavioral tests to generate predictions that otherwise may not be possible with a single source of data. The research will go beyond establishing and explaining individual differences to predicting individual cognitive performance in a variety of tasks.

Machine learning is now a key aspect behind the smart software that is present in many aspects of our daily lives, including face recognition in cameras, chatbots that can answer questions, and speech recognition and language translation on our mobile phones. The prediction models that drive such software are created automatically by machine learning algorithms trained on large amounts of historical data. One issue with these models is that they are often black-box in nature and difficult for humans to understand. In particular, they can be overconfident in their predictions and are not always able to recognize their own limitations. As these types of machine learning models move into more critical tasks, such as autonomous driving and medical diagnosis, it is becoming increasingly important to understand the limitations of such models in real-world practical situations. This research project will address these issues by investigating new mathematical and algorithmic approaches that can improve our ability to assess the performance and confidence of black-box prediction models, particularly when the models are operating in new environments that they have not encountered before. The outcomes of this research will have the potential to significantly improve the reliability and usability of machine learning systems across a broad range of areas such as medicine, transportation, business, and consumer applications.

This project focuses on an aspect of explainable AI concerned with enabling black-box machine learning models (specifically those based on classification and regression) to produce confidence statements about their predictions. This project is pursuing novel methods for understanding the predictions of these models to overcome the implicit overconfidence that otherwise black-and-white, in-or-out classification outcomes can imply. This research project will bring together expertise from cognitive science and computer science in the context of two broad themes. The first theme will focus on developing accurate and robust algorithms that can learn how much confidence to place in a black-box model's predictions. The researchers will investigate new Bayesian calibration methods and develop a broad framework for robust and accurate online assessment of the capabilities of black-box prediction models. The second theme will leverage the algorithmic advances from the first theme to develop new approaches to confidence assessment that can improve the effectiveness of the combined efforts of a black-box predictor and a human decision-maker. This work will in turn provide the basis for trading off prediction accuracy and human effort, and allow for development of techniques that leverage accurate confidence estimates to reduce algorithm aversion and increase trust on the part of the human. Engaging the interest of a broader community will also be a key aspect of the project, with a focus on workshops and hackathons involving under-represented community college students in Southern California, to address broad-ranging questions related to the use of artificial intelligence and machine learning techniques in our everyday world.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.