Comparing OECD PISA Reading in English to Other Languages: Identifying Potential Sources of Non-Invariance

Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals

Y

23

Comparative Analysis; Reading Achievement; Achievement Tests; Secondary School Students; Correlation; Foreign Countries; Culture Fair Tests; Criticism; Cultural Differences; Cross Cultural Studies; Reading Tests; Factor Analysis; Effect Size; Socioeconomic Status; Reading Comprehension; Test Construction; Error of Measurement; Educational Resources; Role; Educational Policy; Item Response Theory; Goodness of Fit

10.1080/15305058.2015.1064431

1530-5058

The use of the Programme for International Student Assessment (PISA) across nations, cultures, and languages has been criticized. The key criticisms point to the linguistic and cultural biases potentially underlying the design of reading comprehension tests, raising doubts about the legitimacy of comparisons across economies. Our research focused on the type and magnitude of invariance or non-invariance in the PISA Reading Comprehension test by language, culture, and economic development relative to performance of the Australian English-speaking reference group used to develop the tests. Multi-Group Confirmatory Factor Analysis based on means and covariance structure (MACS) modeling was used to establish a d[subscript MACS] effect size index for each economy for the degree of non-invariance. Only three wealthy, English-speaking countries had scalar invariance with Australia. Moderate or large effects were observed in just 31% of the comparisons. PISA index of economic, social and cultural status had a moderate inverse correlation with dMACS suggesting that socioeconomic resourcing of education played a significant role in measurement invariance, while educational practice and language factors seemed to play a further small role in non-invariance. Alternative approaches to reporting PISA results consistent with non-invariance are suggested.

As Provided

64

2016

EJ1089565