Lauren Brookman-Frazee - US grants

DESCRIPTION (provided by applicant): The objective of this career development award is to provide the applicant with the advanced training and skills necessary to develop an independent program of research developing methods to translate research- based behavioral interventions for children with autistic spectrum disorders into the public mental health service system. By applying multidisciplinary models of knowledge exchange processes, the applicant will have the skills to develop and test a mental health intervention protocol that individualizes research-based interventions for children with ASD in public mental health settings based on child and family clinical characteristics. Through the training activities, the applicant will (1) develop expertise in the child and family characteristics of school-age children with ASD served in public mental health settings in order to tailor and individualize interventions for this population, (2) obtain training in models of collaborative knowledge exchange from multiple disciplines that will be used in the process of developing a mental health intervention approach based on the research-based interventions for school-age children with ASD and a training model to apply this intervention approach, (3) acquire skills in qualitative and quantitative methodology that can be applied to both examining usual care and for large-scale implementation trials, (4) obtain experience using innovative technology in clinician training, (5) receive training in ethical conduct of scientific research. The research plan that is proposed as part of this career development award is organized in three phases. Phase 1 will focus on characterizing (1) children with ASD and their families served in the public mental health settings in order to individualize research-based interventions for use in these settings and (2) the training needs of mental health service providers from multiple disciplines. Phase 2 will use a collaborative knowledge exchange framework to develop a mental health intervention protocol for children with ASD and a clinician training model for this approach. Phase 3 will involve a feasibility and acceptability study of the implementation of the training model developed. Research under this award will serve as pilot data for future R01 large-scale studies examining the impact on child and family level outcomes of implementing research- based behavioral interventions for children with ASD in public mental health settings.

DESCRIPTION (provided by applicant): The proposed study is a randomized controlled trial of AIM HI, an intervention designed specifically for implementation in community mental health clinics, to reduce problem behavior among school-age children with autism spectrum disorders (ASD). The CMH system plays an important role in caring for school-age children with ASD, especially those who exhibit challenging behaviors. Unfortunately, there are widespread concerns about the quality of care provided to children with ASD in CMH services as therapists lack specialized training to address the complex needs of children with ASD, particularly behavior problems, and do not use EBPs consistently or intensively. Targeting the most common presenting problems has potential to make a significant impact on care for most children with ASD served in CMH clinics. Implementing existing EBPs for ASD in CMH clinics offers an innovative solution to the quality gap in CMH service settings. Our pilot data indicate that clinics and providers are eager to learn how to effectively treat children with ASD. However, existing EBPs were not designed for delivery within CMH clinics. Our previous research began to address the lack of readily implementable EBPs to treat behavior problems in children with ASD. Specifically, we identified and packaged EBP strategies based on the clinical needs of children and training needs of therapists, resulting in the final intervention protocol, An Individualized Mental Health Intervention for Children with ASD (AIM HI). AIM HI is a package of EBP strategies, based on the principles of applied behavior analysis, designed to reduce challenging behaviors in children with ASD ages 5 to 13 served in CMH clinics. Although the individual components of AIM HI are well-established EBP strategies and pilot study data indicate that child behavior problems decrease when their CMH therapists deliver AIM HI, the effectiveness of the protocol has not been established. Therefore, the primary aims are to test the impact of AIM HI on child and family outcomes and determine how child/ family characteristics and level of therapist fidelity moderate treatment effects. Our secondary aim is to collect initial data on implementation outcomes and conduct exploratory analyses to identify barriers and facilitators of AIM HI implementation (e.g., intervention, organization, provider characteristics). Participants will include 29 clinics recruited from San Diego County's publicly-funded service system, 39 program managers, 223 therapists, and 301 child/ parent dyads. The study builds directly on the promising results from the PI's career development award. Our previous studies in this service system provide the empirical, methodological, and collaborative foundation to support the adoption and implementation of the intervention. It also supports the feasibility and validity of the research methods to assess the intervention's fidelity and effectiveness. Consistent with the IACC Strategic Plan for ASD research and Objective 4 of NIMH's Strategic Plan, testing the effectiveness of packages of EBP strategies in usual care settings is hypothesized to ultimately improve treatment of children with ASD in the community.

DESCRIPTION (provided by applicant): Evidence-based practices (EBPs) are increasingly being implemented in large community mental health systems. Much of the research to date has focused on policy, organizational and practitioner variables that predict adoption and initial implementation of EBPs, yet understanding sustainment is crucial to facilitate a return on costly investments in EBP development. Furthermore, most studies focus on the implementation of a single EBP, limiting opportunities to examine theoretically important predictors of implementation success concerning fit between EBPs and the contexts in which they are utilized. The proposed study will examine the impact of a major policy change mandating use of EBPs and organizational context, organizational accommodation, and practitioner attitudes and adaptations as determinants of EBP sustainment in the largest public mental health system in the United States. Our primary outcomes of interest are two aspects of sustainment: practitioner-level penetration and agency and practitioner structural and intervention fidelity to EBP models. Specific aims are as follows: (1) Characterize sustainment outcomes (Penetration, Fidelity); (2) Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment; (3) Identify inner context (organizational and practitioner) and early implementation conditions that determine sustainment outcomes. This project will yield new understanding of whether and how multiple EBPs can be sustained in public mental health systems undergoing a policy-driven community implementation effort. We will produce generalizable models for characterizing sustainment, including feasible and flexible measurement of fidelity across multiple EBPs. The findings will inform the development of implementation interventions to promote sustained delivery of EBPs to maximize their public health impact.

DESCRIPTION (provided by applicant): To improve the overall quality and effectiveness of mental health intervention, it is essential to identify empirically supported behavioral treatments (ESBTs) and assure the accessibility and usability of these treatments for providers in community settings. Fidelity of implementation (FI), or the degree to which an intervention is implemented as was intended, provides necessary information for training and maintaining providers' effectiveness. However, the tools and procedures used to measure FI and provide feedback to providers must be effective and efficient. Through archival data analysis and prospective assessment of individual treatment components, we aim to develop and assess an FI instrument and assessment procedures for an ESBT for children with autism. Autism is a key area of concern in the field of child mental health; Pivotal Response Training (PRT) is a naturalistic ESBT for this population (Humphries, 2003; National Standards Project, 2009). PRT is routinely used in community settings with limited data regarding treatment integrity or effectiveness (Stahmer, Collings & Palinkas, 2005). PRT fidelity of implementation (FI) measures were developed for use in research settings and have received minimal validation and adaptation for use by community therapists. The specific components of PRT have not been independently evaluated to determine how FI relates to child outcome, and the effect of ancillary factors such as therapist discipline, setting, education, affect, and the treatment environment on treatment integrity and child outcome have not been examined. The overall goal of this project is to inform strategies for enhancing fidelity of PRT in community service settings Toward this goal, there will be six specific aims: 1) Evaluate relationships between FI of PRT and within-session child skills and behavior via archival data analysis. 2) Evaluate relationships between provider characteristics and FI of PRT components via archival data analysis. 3) Individually test possible modifications to components of PRT using deconstructive evaluation. 4) Develop and evaluate a PRT FI instrument and assessment procedures in partnership with community providers. 5) Evaluate effectiveness of a PRT FI instrument and assessment procedures including (a) examining improvements to training and supervision methods using the instrument and (b) analyzing possible moderating variables affecting FI of PRT and child outcome in a prospective study. 6) Prospectively assess the relationship between provider implementation of PRT and child skills and behavior. The proposed scientific model will generate data and important products supporting ESBT use in the autism services community. This research will also serve as a general model for evaluation and improvement of FI knowledge and practice for the mental health services community and implementation science.

DESCRIPTION (provided by applicant): Evidence-based practices (EBPs) are increasingly being implemented in large community mental health systems. Much of the research to date has focused on policy, organizational and practitioner variables that predict adoption and initial implementation of EBPs, yet understanding sustainment is crucial to facilitate a return on costly investments in EBP development. Furthermore, most studies focus on the implementation of a single EBP, limiting opportunities to examine theoretically important predictors of implementation success concerning fit between EBPs and the contexts in which they are utilized. The proposed study will examine the impact of a major policy change mandating use of EBPs and organizational context, organizational accommodation, and practitioner attitudes and adaptations as determinants of EBP sustainment in the largest public mental health system in the United States. Our primary outcomes of interest are two aspects of sustainment: practitioner-level penetration and agency and practitioner structural and intervention fidelity to EBP models. Specific aims are as follows: (1) Characterize sustainment outcomes (Penetration, Fidelity); (2) Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment; (3) Identify inner context (organizational and practitioner) and early implementation conditions that determine sustainment outcomes. This project will yield new understanding of whether and how multiple EBPs can be sustained in public mental health systems undergoing a policy-driven community implementation effort. We will produce generalizable models for characterizing sustainment, including feasible and flexible measurement of fidelity across multiple EBPs. The findings will inform the development of implementation interventions to promote sustained delivery of EBPs to maximize their public health impact.

12. PROJECT SUMMARY The Centers for Disease Control (CDC) estimates that 1 in 68 children have autism spectrum disorder (CDC, 2014) and the annual cost of ASD in the U.S. is estimated to be $236 billion. Evidence-based interventions have been developed and demonstrate effectiveness in improving child outcomes. However, research on generalizable methods to scale up these practices in the multiple service systems caring for these children has been limited and is critical to meet this growing public health need. We propose to conduct two, coordinated studies testing the effectiveness of the ?Translating Evidence-based Interventions (EBI) for ASD: Multi-Level Implementation Strategy? (TEAMS) model. TEAMS focuses on improving implementation leadership, organizational climate, and provider attitudes and motivation in order to improve two key implementation outcomes ? provider training completion and ASD EBI fidelity, and subsequent child outcomes. The TEAMS? LEAD module applies the LOCI (?Leadership and Organizational Change for Implementation?)1 strategies, and the TEAMS-PROV module applies MI (Motivational Interviewing) strategies to facilitate individual provider and organizational behavior change. These studies will use a randomized implementation/effectiveness Hybrid, Type 3, trial. Study #1 will test the TEAMS model with the AIM HI intervention (?An Individualized Mental Health Intervention for ASD?) in publicly-funded mental health services. Study #2 will test TEAMS with the CPRT intervention (?Classroom Pivotal Response Teaching?) in education settings. AIM HI and CPRT data indicate that (1) provider attitudes towards EBI and (2) implementation leadership are promising targets of implementation interventions. TEAMS will target these specific mechanisms of change. This study will randomize 37 MH programs and 37 school districts to one of four groups (TEAMS PROV+LEAD; TEAMS- LEAD; TEAMS-PROV; EBI training only) to test the effectiveness of combining standard, EBI-specific training with the two TEAMS modules individually and together on multiple implementation outcomes. A dismantling design will be used to understand the effectiveness of TEAMS and the mechanisms of change across settings and participants. Implementation outcomes including provider training completion, fidelity and child behavior change will be examined for 295 MH providers, 295 teachers and 590 children (combined across studies). This implementation intervention has the potential to increase quality of care for ASD in publicly-funded settings by improving effectiveness of EBI implementation. The process and modules will be generalizable to multiple service systems, providers, and interventions, providing broad impact in community services.

PROJECT ABSTRACT Drs. Brookman-Frazee and Lau (R01MH112536-01 Fidelity measures are largely impractical for assessing psychotherapy quality in usual care (UC) because they: (1) often involve labor-intensive observational methods, (2) measure many psychotherapy features with unknown validity for predicting care outcomes, and (3) are almost exclusively EBP-specific, thereby limiting their utility in system reform efforts that involve implementing multiple EBPs to meet the range of patients presenting for services.1 Therapist reports of EBP delivery represent a pragmatic approach to QA. Despite concerns about the validity of therapist reports,2 emerging findings suggest that therapist reports of EBP delivery align with other treatment integrity measures when EBPs are delivered and predict variance in client-level outcomes, supporting their potential use as a QA strategy in the context of EBP implementation in UC.3,4 We propose to build upon our current 4KEEPS R01 study (R01MH100134) to develop a pragmatic QA of ongoing multiple EBP delivery in children?s mental health services. The purpose of this ongoing study is to characterize multiple EBP sustainment and identify organizational and therapist characteristics that predict sustainment outcomes within the nation?s largest public mental health system.5 Our community partner, the Los Angeles County Department of Mental Health (LACDMH), is the nation?s largest public mental health (MH) authority, and is a leader in implementing multiple EBPs in system driven reform. However, LACDMH leaders are struggling with lack of tools to feasibly assess ongoing quality of multiple EBPs. EBP concordant care, a primary sustainment outcome referring to the degree to which a therapist?s practice resembles the essential strategies one would expect within an EBP protocol for a given MH focus, offers a potential solution to this problem. We depart from protocol-specific fidelity measures that exclusively assess features of one specific treatment by assessing elements of sets or families of EBP?s for target MH conditions. We have developed the EBP Concordant Care Assessment (ECCA) to provide a common metric to assess therapist delivery of EBPs for major child MH targets. Item content was drawn from established tools used to characterize UC therapist practice6-8 and EBP developers? report of core strategies essential to treatment integrity. Our preliminary data support the promising validity of the innovative ECCA to assess therapy content and highlight essential refinement needed to improve assessment of therapeutic techniques. It is also essential to isolate which content and technique strategies function as quality indicators predictive of client outcomes. In the proposed study, we will use a community-partnered approach to build upon the initial ECCA (?ECCA-??) for pragmatic QA in UC. We will: (1) modify instrumentation to increase concordance between therapist report and gold standard observational ratings (?ECCA-??), (2) identify a subset of ECCA-? items that predict client-level outcomes, and (3) provide guidelines for administration and interpretation. The results will inform future research aimed to develop and test an ECCA Toolkit for use by agencies and systems to monitor and improve the quality of ongoing multiple EBP delivery.

12. PROJECT SUMMARY The Centers for Disease Control (CDC) estimates that 1 in 68 children have autism spectrum disorder (CDC, 2014) and the annual cost of ASD in the U.S. is estimated to be $236 billion. Evidence-based interventions have been developed and demonstrate effectiveness in improving child outcomes. However, research on generalizable methods to scale up these practices in the multiple service systems caring for these children has been limited and is critical to meet this growing public health need. We propose to conduct two, coordinated studies testing the effectiveness of the ?Translating Evidence-based Interventions (EBI) for ASD: Multi-Level Implementation Strategy? (TEAMS) model. TEAMS focuses on improving implementation leadership, organizational climate, and provider attitudes and motivation in order to improve two key implementation outcomes ? provider training completion and ASD EBI fidelity, and subsequent child outcomes. The TEAMS? LEAD module applies the LOCI (?Leadership and Organizational Change for Implementation?)1 strategies, and the TEAMS-PROV module applies MI (Motivational Interviewing) strategies to facilitate individual provider and organizational behavior change. These studies will use a randomized implementation/effectiveness Hybrid, Type 3, trial. Study #1 will test the TEAMS model with the AIM HI intervention (?An Individualized Mental Health Intervention for ASD?) in publicly-funded mental health services. Study #2 will test TEAMS with the CPRT intervention (?Classroom Pivotal Response Teaching?) in education settings. AIM HI and CPRT data indicate that (1) provider attitudes towards EBI and (2) implementation leadership are promising targets of implementation interventions. TEAMS will target these specific mechanisms of change. This study will randomize 37 MH programs and 37 school districts to one of four groups (TEAMS PROV+LEAD; TEAMS- LEAD; TEAMS-PROV; EBI training only) to test the effectiveness of combining standard, EBI-specific training with the two TEAMS modules individually and together on multiple implementation outcomes. A dismantling design will be used to understand the effectiveness of TEAMS and the mechanisms of change across settings and participants. Implementation outcomes including provider training completion, fidelity and child behavior change will be examined for 295 MH providers, 295 teachers and 590 children (combined across studies). This implementation intervention has the potential to increase quality of care for ASD in publicly-funded settings by improving effectiveness of EBI implementation. The process and modules will be generalizable to multiple service systems, providers, and interventions, providing broad impact in community services.