Impact of Design Effects in Large-Scale District and State Assessments

Y

15

State Programs; Sampling; Research Design; Error of Measurement; Testing Programs; Statistical Significance; School Districts; Probability; Sample Size; Response Rates (Questionnaires); Weighted Scores; Item Analysis; Equated Scores; Item Response Theory; Testing Problems; Evaluation Methods; Evaluation Problems; Experimenter Characteristics; Test Reliability; Test Validity; Group Testing

10.1080/08957347.2014.973561

0895-7347

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling errors, in turn, results in unanticipated instability in the testing program and an increase in Type I errors in significance tests. This is especially true when the standard error of equating is underestimated. The problem is caused by the typical district and state practice of using nonprobability cluster-sampling procedures, such as convenience, purposeful, and quota sampling, then calculating statistics and standard errors as if the samples were simple random samples.

As Provided

21

2015

EJ1048933