Providing Comparability of Test Results for Scenario-Based Assessment

The purpose of the paper is to provide comparability evidence of scenario-based assessment. The use of scenario-based tasks for measuring complex constructs has become a modern direction of educational and psychological testing, which sets new practical and research tasks, including checking the comparability of SBTs forms. However, the methodology of comparability of SBTs is underdeveloped and there is a shortage of research answers this issue. In this study, we used test data from computerized SBT aimed at measuring critical thinking of primary school children. The research design was developed considering the age of target sample. The confirmatory factor analysis (CFA) was applied to examine the comparability of forms. In order to provide comparability evidence of SBT forms methodology of invariance testing with CFA was used. Configural and strong (equal factor loadings and thresholds) invariance with categorical variables was tested across task forms along with the mean comparison of all latent factors. The study employs CFA bi-factor model with two main factors related to components of critical thinking (analysis and conclusion) and two additional uncorrelated factors to explain the common context for indicators. Findings indicate that factor structure show a good fit for both task forms. Also, forms are invariant with respect to factor loadings and partly to thresholds. Thus, forms can be considered comparable on the main factors related to critical thinking. The context of the scenario probably affects the subjects' perception of the scenario and their observed behavior during the test, but not their critical thinking score. Based on the results of the paper, practical recommendations are offered to improve the instrument.