Guanglei Hong - US grants

This research project will develop new strategies for investigating causal mechanisms in multi-site experimental and quasi-experimental evaluations of intervention programs. Past research in a wide range of fields often has reported considerable cross-site heterogeneity in the average program effect. However, program evaluators have not been able to take full advantage of multi-site data. The analytic strategies to be developed in this project will be suitable for testing scientific theories about the intermediate process of an intervention and for assessing whether and how the program mechanism operates differently across settings. The new strategies will allow applied researchers to flexibly ask a new set of empirical questions crucial for testing the generalizability of an intervention theory across settings and for unpacking complex mediation mechanisms, making it possible to revisit the intervention theory and to suggest specific site-level modifications of the intervention practice. This project is motivated by application examples, including the National Job Corps Study, which evaluates a program designed to promote economic independence of disadvantaged youths; the Head Start Impact Study, which evaluates the federal early childhood education program for children living in poverty; and math curricular reforms in Chicago Public Schools aimed at improving the math learning of low-achieving ninth graders. The research team will provide open-source R packages and Stata ado files along with user manuals and pedagogical data examples for dissemination.

This research is built on the tradition of mediation analysis in the social sciences and on the recent developments in statistical theories and methods for causal mediation analysis. The existing methods, in general, require the analyst to specify both mediator models and outcome models involving comparatively strong model-based assumptions. Assessing between-site heterogeneity in causal mechanisms is challenging especially when many of the model-based assumptions are impractical. The research team has made initial attempts to extend a propensity score-based weighting method to multisite analysis of simple mediation mechanisms. This weighting method does not require outcome model specifications. The latest analytic procedure, however, is limited to experiments with a relatively large sample size per site. To overcome the existing challenges, this project will advance methodological developments in causal parameter estimation, propensity score estimation, and sensitivity analysis. The investigators will develop a novel pseudo-outcome random-effects strategy for causal parameter estimation analyzed through method-of-moments to avoid model-based assumptions. By pooling data from all the sites, the analysis will be unconstrained by a small sample size per site and will be flexible for investigating complex mediation mechanisms. Asymptotic variances will be obtained that take into account uncertainty in propensity score estimation. For enhancing robustness to propensity score model misspecification, a covariate-balancing propensity score estimation approach will be incorporated. The investigators also will develop a new weighting-based approach to sensitivity analysis for omitted post-treatment as well as pre-treatment confounders. These new strategies will be extended from simple to complex mechanisms and from experimental to quasi-experimental multi-site data.

The goals of this capacity building institute are (a) to introduce participants to rigorous and novel applications of advanced methods to STEM education research, (b) to provide them with continuous methodological support in research planning, data analysis, and publication, and (c) to create a community that prepares emerging scholars for taking leadership in advancing STEM education research. This capacity building award initiates an institute in quantitative and computational methods for STEM education research through a collaboration between researchers at the University of Chicago and those at Michigan State University. Through launching a national campaign of recruitment, the team will select a diverse cohort of NSF Fellows of STEM Education Research among early- and mid-career scholars, especially those from under-represented backgrounds.

Instructors and Fellows will critically evaluate existing STEM education research throughout six courses over the summers in year 1 and year 2. Institute training will focus on (1) research designs and causal inference, (2) measurement, (3) social network analysis, (4) multilevel modeling, (5) causal mediation analysis, and (6) computational methods for analyzing qualitative and social media data. Structured discussions of Fellows? ongoing studies will be supplemented by guest speaker presentations and round-table discussions. Fellows will present their capstone projects and further develop research collaborations in the summer of year 3. A customized website will disseminate to the broader STEM education research community.

This research institute is funded by ECR (the core research program of the Education and Human Resources directorate) and attends specifically to critical aspects in research capacity building.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.