Bryce D. McLeod - US grants

DESCRIPTION (provided by applicant):The main goals of the proposed study are to establish the psychometric properties of the Therapy Process Observational Coding System for Child Psychotherapy (TPOCS) and to assess applications of the TPOCS to the study of therapy process and outcome. The TPOCS is being developed to measure specific (e.g., therapeutic interventions) and nonspecific (e.g., therapeutic alliance) process variables that are prominent in child psychotherapy but for which observational assessment has not been well-developed to date. Development of the TPOCS will provide the field with a new research tool to quantify theoretically important processes in child psychotherapy and to assess proposed mediators of child psychotherapy outcome. The psychometric properties of the TPOCS will be assessed using a sample of treatment-as-usual psychotherapy tapes drawn from a larger ongoing study. Once the psychometric properties of the TPOCS are established, the TPOCS scales will be used to investigate three key questions of theoretical interest in therapy process research: (a) Are some therapy processes (i.e., specific and non-specific) associated with therapy engagement? (b) Do some therapy processes predict changes in children?s post-treatment outcomes? (c) Are the linkages between therapy processes and children?s posttreatment outcomes mediated by therapy engagement?

DESCRIPTION (provided by applicant): The ultimate objective of the proposed study is to further develop a set of measures capable of assessing the integrity of cognitive-behavioral treatments (CBT) for anxiety in youths. Despite the success of CBT for child anxiety in controlled trials, measurement gaps in the field limit efforts to (a) refine and optimize the effectiveness of CBT, and (b) evaluate the success of CBT clinician training efforts. Indeed, there is a paucity of evidence-based measurement strategies to assess the integrity of efforts to implement CBT programs for child anxiety and none that incorporate the three dimensions of integrity. The proposed study addresses this significant methodological gap through the further development and validation of a set of observational measures designed to assess the key components of treatment integrity-treatment adherence, therapist competence, and treatment differentiation-concerning CBT for youths with anxiety. The project will accomplish two primary aims: (a) Establish reliability (interrater reliability, internal consistency, generalizability) and validity (convergent, discriminant) for the treatment integrity measures;and (b) Provide evidence linking data from these measures to (i) child outcomes, and (ii) therapy input variables (e.g., client, therapist characteristics). Recorded therapy sessions from three randomized clinical trials (RCTs;efficacy and 2 effectiveness trials) evaluating CBT for youth with primary anxiety disorders will be used to accomplish these aims. The research project progresses through three phases. In Phase I, the psychometric properties of the treatment integrity measures will be assessed using recorded therapy sessions of CBT for anxiety in youths from one efficacy and one effectiveness trial. The psychometric data provided in Phase I will be used to refine the measures during Phase II. The measures will then be used in Phase III to code archived therapy sessions of CBT for anxiety in youths from an effectiveness study. The resulting data will be used to investigate key questions of theoretical and practical interest, with an eye toward optimizing the effectiveness and efficiency of CBT, including: (a) Exploring the relations among treatment adherence, competence, differentiation, and outcomes in CBT for anxiety in youth;and (b) Identify sources of variation in the integrity of CBT for anxiety in youth. The proposed study is the first to examine the relation between the three integrity components (treatment adherence, competence, differentiation) and clinical outcomes in CBT for children with anxiety disorders. Moreover, the measures produced by the proposed study may be used as a gold standard against which to validate parent- and therapist report measures of integrity.

? DESCRIPTION (provided by applicant): The ultimate objective of the proposed study is to develop a pragmatic observational treatment integrity instrument designed to support quality control efforts in community settings that strikes a balance between practicality (short, easy to use) and validity evidence (sensitivity to change, criterion, predictive). Practical instruments tht have validity evidence for use in community settings are needed to support implementation research. This is because most existing instruments of constructs central to implementation research are impractical (too long, too difficult to use) and lack validity evidence for use in community settings (sensitivity to change, criterion, predictive). However, as few pragmatic instruments exist it is not yet known if it is even possible to develop instruments that are both practical and possess strong validity evidence. The proposed study addresses this measurement gap and, in so doing, evaluates a critical question pertaining to implementation science: Can pragmatic instruments that balance validity evidence and practicality be developed? The project will accomplish two primary aims: (a) Develop the Treatment Integrity Measure- Efficient (TIME) a short observational treatment integrity instrument; and (b) Optimize the practicality and validity evidence of the TIME Scale scores. For Aim 1, the TIME will be developed from existing observational treatment integrity instruments that have strong evidence for their valid use in efficacy and effectiveness contexts. The TIME will have three scales that represent key treatment integrity domains relevant to quality control efforts: adherence, differentiation, and competence. The validated source instruments have been used to code sessions from an efficacy trial and 2 effectiveness trials (N = 144 youth participants; N = 1521 therapy sessions). Using Rasch measurement models we will use these data to develop a short observational instrument (~12 items) from the three validated source instruments containing 87 items that has a small number of items for each of the three TIME Scales (Adherence, Differentiation, Competence). For Aim 2, the TIME will be used to code sessions from the 3 trials (1 efficacy trial and 2 effectiveness trials) previously coded with the three source observational treatment integrity instruments. Each session will be coded for 5, 10, 15, and 20 minutes with the TIME. For each of the three TIME Scales (Adherence, Differentiation, Competence), the previous archived scores from coding the full session will be used as benchmarks to determine the shortest observation length (5, 10, 15, 20 minutes) that retains preferred reliability and validity features (Aim 2). The end product, TIME, will be a pragmatic/practical observational treatment integrity instrument designed as a quality control tool with validity evidence that is ready to be field-tested in community settings by end-users to determine the instrument's utility for implementation science.

Project Summary The objective of the proposed study is to develop implementation tools to facilitate an improved quality of care for school-aged youth with autism spectrum disorder (ASD) receiving services in community mental health settings. The NIMH Strategic Research Priorities call for innovative service delivery models to improve the outcomes of mental health services received in diverse communities and populations, including the use of technology-based strategies to make evidence based practices (EBPs) available, deliverable, effective, and scalable in non-specialty settings, and for service system interventions designed to improve functional outcomes for people with ASD (Objective 4). To achieve these goals, EBPs for school-aged youth with ASD need to be prepared and optimized for implementation in the community settings where most youth with ASD receive treatment. Implementation of EBPs for youth with ASD will require effective Quality Control (QC) procedures (therapist training and support procedures and QC instruments to gauge the success of the training/support procedures). As a first step towards these goals, in the proposed project we will accomplish three primary aims: (a) Develop a flexible, internet-based, video supported training and real time therapist guidance application to support clinician implementation of the Modular EBPs for Youth with ASD (MEYA), a compendium of EBPs for school-aged youth with ASD; (b) Develop the MEYA Integrity Scale, a pragmatic treatment integrity scale designed to gauge the success of the MEYA Training System and serve as a feedback tool in supervision; and (c) Assess the feasibility and promise of these MEYA QC procedures to improve therapist delivery of MEYA via key mediators (therapist knowledge, engagement) in a multiple baseline study. A panel of stakeholders comprised of end-users from community treatment centers for youth with autism will advise the research team on design features of the MEYA Training System and Integrity Scale. A software designer and videographer will work with the research team to realize the MEYA Training System, which will include multiple design features to maximize clinician engagement, motivation, and learning guided by contemporary models of implementation and learning (e.g., Proctor et al., 2011). The MEYA Integrity Scale will be refined and validated against a pre-existing archive of 1020 MEYA and comparison treatment sessions from NIH-funded clinical trials. The multiple baseline study will recruit 15 clinicians working with youth with ASD in community settings. The MEYA Training System will be assessed for acceptability and feasibility in this trial, and the primary outcome measure will be therapists' scores on the MEYA Integrity Scale following the onset of the intervention. The MEYA Training System and Integrity Scale will be refined based on stakeholder and clinician feedback and prepared for evaluation in a future R01 effectiveness trial testing the MEYA Training System in conjunction with different training and supervision approaches typically used in community settings.

PROJECT SUMMARY Evidence-based programs (EBPs) delivered in school settings show great promise in reducing risk for the development of emotional and behavioral disorders (EBDs) among elementary-aged children. However, efforts to sustain EBPs in school settings face a number of barriers. Improving EBP sustainment in schools thus represents a public health priority, but very little research exists to inform the development of sustainment strategies. In order to address this gap, and build a foundation for sustainment strategy development, the specific factors that predict sustainment need to be identified. To achieve this goal, the proposed project will accomplish three primary aims: (1) Determine if malleable teacher (self-efficacy, burnout, attributions), intervention (usability), and school (climate, classroom atmosphere, classroom-level adversity) factors predict EBP treatment fidelity (adherence, competence) and modifications to EBPs (fidelity-consistent, fidelity- inconsistent) during implementation or sustainment; (2) Assess the impact of EBP treatment fidelity and EBP modifications on child outcomes (disruptive behavior problems, social skills) during implementation and sustainment; and (3) Explore the mechanisms through which teacher, intervention, and school factors influence sustainment outcomes. This project builds on a 4-year, federally-funded randomized controlled trial evaluating BEST in CLASS, an effective, teacher-delivered intervention for K-3rd grade children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools in diverse communities located in VA and FL. During the implementation phase, the teachers randomized to BEST in CLASS will receive a one-day training followed by 14 weekly sessions of practice-based coaching. To achieve study aims, a multilevel, interrupted time series design will be used to examine the relation between baseline factors, treatment fidelity (adherence and competence to the core BEST in CLASS practices), modifications (fidelity- consistent, fidelity-inconsistent), and child outcomes (disruptive behavior, social skills), followed by a mixed method approach to elucidate the mechanisms that influence sustainment outcomes. Independent observers will use a gold-standard, observer-rated measure with strong psychometric properties to rate treatment fidelity and modifications to BEST in CLASS at 8 timepoints during implementation (baseline, 4 weeks, 8 weeks, post- intervention [14 weeks]) and sustainment (baseline, 4 weeks, 8 weeks, 14 weeks). Our long-term goal, consistent with NIMH Strategic Priority 4 to strengthen the public health impact of funded research, is to engage in research to maximize the sustainment of EBPs in school settings and thus improve outcomes for children. This R01 is a critical building block in our effort to realize the promise of EBPs and will help us create the blueprint for a strategy designed to improve EBP sustainment in schools that we can develop with a NIMH R34 proposal and subsequently evaluate with a R01 proposal.