Computerized Adaptive Testing for Cognitive Diagnosis in Classroom: A Nonparametric Approach

Abstract

The Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) has been suggested by researchers as a diagnostic tool for assessment and evaluation. While model-based CD-CAT is relatively well-researched in the context of large-scale assessments, this type of system has not received the same degree of development in small-scale settings, where it would be most useful. The main challenge is that the statistical estimation techniques successfully applied to the parametric CD-CAT require large samples to guarantee the reliable calibration of item parameters and accurate assignments of examinees. In response to the challenge, a nonparametric approach that does not require any parameter calibration, and thus can be used in small educational programs, is proposed. Unlike other CD-CAT algorithms, the proposed nonparametric CD-CAT uses the nonparametric classification (NPC) method to assess and update the student's ability state while the test proceeds. Based on a student's responses, possible proficiency classes are identified, and items which can discriminate them are chosen next. The simulation results show that the proposed nonparametric item selection (NPS) method outperformed the compared parametric CD-CAT algorithms and the differences were more significant when the item parameter calibration was not optimal.

The Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) has been suggested by researchers as a diagnostic tool for assessment and evaluation. While model-based CD-CAT is relatively well-researched in the context of large-scale assessments, this type of system has not received the same degree of development in small-scale settings, where it would be most useful. The main challenge is that the statistical estimation techniques successfully applied to the parametric CD-CAT require large samples to guarantee the reliable calibration of item parameters and accurate assignments of examinees. In response to the challenge, a nonparametric approach that does not require any parameter calibration, and thus can be used in small educational programs, is proposed. Unlike other CD-CAT algorithms, the proposed nonparametric CD-CAT uses the nonparametric classification (NPC) method to assess and update the student's ability state while the test proceeds. Based on a student's responses, possible proficiency classes are identified, and items which can discriminate them are chosen next. The simulation results show that the proposed nonparametric item selection (NPS) method outperformed the compared parametric CD-CAT algorithms and the differences were more significant when the item parameter calibration was not optimal.