The Effects of Incomplete Q-Matrix on Parameter Estimates and Classification Accuracy in the DINA and DINO Models and Non-parametric Approach

Abstract

用Q矩陣為基礎以側寫考生認知的認知診斷模型越來越受關注，故確保認知辨識性的Q矩陣完備性很重要。然而，要準備一份具有Q矩陣完備性的測驗往往是很困難的，尤其是當感興趣的能力數量很大時。因此，本文主要的目的是探究不完備Q矩陣對認知辨識率的準確度和認知診斷模型的參數估計的影響。我們探究了不完備Q矩陣對DINA/DINO模型和無母數方法的影響。透過模擬研究不同設定下不完備Q矩陣的影響。在這三個模型的認知辨識率上使用等價分類的概念探究不完備Q矩陣的影響。模擬結果顯示，不完備Q矩陣的影響並不如我們預期的嚴重，特別是在即使有完備Q矩陣亦無法準確分類的狀況。在認知辨識率的比較，參數模型在面對不完備Q矩陣時更具穩健性。且不完備的Q矩陣對參數模型的題目參數估計沒有顯著影響。There has been growing interest in Q-matrix based cognitive diagnosis models to assess examinees’ attribute profiles. The completeness of Q-matrix is important for assuring the identification of all attribute profile classes. However, it is often difficult to have assessments with complete Q-matrix especially when the number of attributes of interest is large. The main objective of this research it to study the effects of incomplete Q-matrix on the classification accuracy of examinees’ attribute profiles and on the parameter estimates for the cognitive diagnosis models. We investigate the effects of incomplete Q-matrix in the DINA/DINO models and the non-parametric method suggested by Chiu& Douglas (2013). Simulation studies are carried out to study the effects of incomplete Q-matrix under different scenarios. The idea of the equivalence class is used on the classification accuracy for these three models to explore the effect of incomplete Q-matrix. Our results show that the effects of incomplete Q-matrix were not as formidable as we expected, especially for the cases where the imprecise classification will happen even with complete Q-matrix. As for the classification accuracy, the parametric model is more robust than the non-parametric approach. Moreover, incomplete Q-matrix did not have a significant effect on the maximum likelihood estimation of item parameters.