Estimating Cross-Site Impact Variation in the Presence of Heteroscedasticity

Y

6

Research Methodology; Policy; Evaluation Research; Randomized Controlled Trials; Program Evaluation; Models; Computation; Bias; Statistical Significance; Simulation; Statistical Analysis; Research Design; Data Collection; Data Analysis

To date, evaluation research and policy analysis have focused mainly on average program impacts and paid little systematic attention to their variation. Recently, the growing number of multi-site randomized trials that are being planned and conducted make it increasingly feasible to study "cross-site" variation in impacts. Important technical questions arise when one uses data from a multi-site randomized trial to estimate the magnitude of cross-site impact variation and assess the statistical significance of these estimates. Some of these questions arise from the likelihood that variation in "individual" outcomes (which in the context of multi-level models is often referred to as a "level-one residual variance" or "residual variance" for short) might be different for the treatment group and control group members and/or might be different for different sites. In addition, to the extent that different populations and local conditions are represented by different sites, the variance of individual "counterfactual" outcomes (i.e. potential outcomes under no treatment) and therefore the variance of control group members' outcomes will differ across sites. This situation can be exacerbated if "the extent of" individual variation in program impacts also varies across sites because of their differences client populations and local conditions. These "differences in individual outcome variation" represent forms of heteroscedasticity which the authors refer to as: (1) T/C heteroscedasticity; and (2) cross-site heteroscedasticity. In this study, the following methodological questions are addressed through a series of simulations: To what extent are estimators of cross-site impact variation biased and tests of statistical significance incorrect when one does not take into account how the variance of individual outcomes differs between treatment and control group members and/or across sites? In other words, in practice, what are the risks of "estimating too few" residual variances and how does this risk vary under a wide range of conditions? (2) Among different estimators that take heteroscedasticity into account--estimators based on either a random-effects model, on a mixed-effects model or on no model (a method of moments estimator)--how do they compare? and (3) What recommendations can be made for practice?

ERIC

10

2016

ED563208