Goal of research
The objective of this project is twofold. The first is to evaluate the learning outcomes of engineering students in the countries participating in the study using Item Response Theory models, taking into account collateral information (particularly, item response times); the second is to identify the factors of career outcomes (including the attitudes to dishonesty in the workplace) of graduates who graduated from electronic engineering and computer sciences in Russian universities. Four tasks encompass these objectives:
1) to provide validity and reliability evidence of the second wave of testing of Russian, Chinese and Indian students using Item Response Theory machinery;
2) to prove the quality of the data and to provide comparability for the results of measurement of the professional competences among Russian, Chinese and Indian engineering students using Item Response Theory machinery and item response times as a collateral information;
3) to assess the relationship between students' attitudes towards academic dishonesty (plagiarism and cheating on exams) at university and their attitudes towards dishonesty in the context of post-graduate employment;
4) to identify the factors of graduates’ career outcomes.
In the field of psychometric work (tasks 1 and 2) used mathematical and statistical apparatus of Item Response Theory (IRT) recasted in terms of Generalized Linear Mixed Models and Cross-Classified Multiple Membership models, in particular the methodology of explanatory IRT models from the Rasch modeling paradigm. To accomplish the tasks 3,4 we used the descriptive analysis, multilevel regression modelling, and factor analysis.
Empirical base of research
To compare the learning outcomes (and to ensure the validity and comparability of the measurement of learning outcomes across Russia, China and India, tasks 1 and 2) we used the data on learning outcomes in the disciplines of fundamental cycle (mathematics and physics) of Russian and Chinese students gathered in 2015-2017, as well as data on learning outcomes of Indian students collected in 2017-2019 (longitudinal measurement design: two cohorts – 1st and 3rd graders with follow-up testing after two academic years). To introduce item response times in the IRT model (task 2), we used data on item response times gathered during the second wave of testing (measurement of the professional competences). This was made possible due to utilizing online platform for test administrating, which recorded response times. All the samples are clustered and randomized (university as a cluster) and nationally representative. For the analysis in tasks 3 and 4, we used the survey data collected among the research participants from Russia eight months after they graduated from universities (Spring, 2018).
Results of research
The analysis of validity and comparability of measurements of learning outcomes of students in Russia, China and India (task 1) showed that the collected data can be used to compare the learning outcomes of engineering students in these countries. Moreover, this data can be used for the analysis of educational progress cross-nationally.
Results of the IRT-analysis of response times as a collateral information demonstrate that the data with response times can be used for comparison of educational achievements of engineering students cross-nationally across Russia, China and India. Although introduction of item response times as a collateral information in IRT model changes latent space of item parameters, it is possible to establish cross-national comparability of results. Moreover, utilizing of such information allows for the sufficient increase in reliability estimates of measurement.
Descriptive analysis of the graduates’ attitudes towards dishonesty in the context of working life has shown that more than half of participants (61-62%) tolerate taking credits for others’ work and lying about work quality (task 3). Students of Moscow and St. Petersburg universities are more inclined to tolerate taking credits for others’ work than students who graduated from regional universities. The study of the relationship between students' attitudes towards academic dishonesty at university and their attitudes towards dishonesty in the workplace has shown a significant correlation between these parameters controlling for individual characteristics of students, the characteristics of their university, as well as the parameters of their current work.
The study of the factors of graduates’ career outcomes (task 4) has shown that the differences in the engineering graduates’ wages may be explained by both their skills (critical thinking) and by the type of university they graduated from, which testifies in favor of both the theory of human capital and the signal theory. The main factor predicting graduates’ wages is the full-time work after graduation. The analysis of the factors of graduation outcomes has shown that the employment of postgraduates during their studies has a negative impact on their probability of defending the thesis.
Level of implementation, recommendations on implementation or outcomes of the implementation of the results
The methodological solutions obtained in this study can be scaled to other projects and research areas for enhancement of methodological sides of the projects. The obtained empirical results can be used to improve the functioning of the higher education system.