Evaluation of Mean and Covariance Structure Analysis Model in Detecting Differential Item Functioning of Polytomous Items: in Comparison with GMH, and Poly-SIBTEST

Abstract

Item bias is a well-known and important issue in educational testing and therefore the investigation of Differential Item Functioning (DIF) has long been proven valuable in evaluating the fairness or quality of an item. In this thesis, a simulation study was conducted for polytomous responses to evaluate the efficacy of using the multiple-group Structural Equation Model (SEM) to detect DIF, in comparison with two nonparametric DIF indices, Poly-SIBTEST and Generalized Mantel-Haenzel (GMH) methods. The multiple group SEM model was used to generate data with five ordered response categories. Items exhibiting DIF were modified by allowing their threshold parameters to differ between the focal and reference groups. Five factors were manipulated: three test lengths (10, 20, 30), three sample sizes (500, 1000, and 3000), two ability distributions, four percentages of DIF items (0%, 10%, 20%, and 30%), and four different size proportions of the focal and reference groups (80/20, 70/30, 60/40, and 50/50), to produce all the conditions of the data sets. Each condition was replicated 100 times to facilitate Type I error and Power calculations of the three detection procedures under consideration. Our results suggested that the DIFFTEST approach under MG-MACS to DIF detection had the smallest overall Type I error rate and comparable or higher Power than GMH and Poly-SIBTEST in all conditions. When polytomous data were generated under the MG-MACS model, the DIFFTEST procedure was viable for DIF detection even for tests with as few as ten items. In conclusion, DIFFTEST performed the best for detecting DIF in polytomous items while comparing to GMH and Poly-SIBTEST, for yielding lower Type I error rates and higher Power for different types of DIF conditions.