Six tasks encompass this objective:
to ensure the validity and cross-national comparability of instruments used to measure student educational outcomes in Russia, China and India;
to compare the student learning outcomes across Russia, China and India;
to build the special Item Response Theory models to analyze the students learning progress in mathematics in Russia during the university studies;
to identify the institutional- and individual-level factors related to the learning progress of Russian students in mathematics;
to collect survey data on the career paths of engineering graduates who previously participated in the study (in 2015 and 2016) in Spring, 2018;
to describe the characteristics of employment of engineering graduates and their relation to the characteristics of their educational experience at university.
In the field of psychometric works (tasks 1 and 3) we applied statistical machinery of Item Response Theory, in particular the paradigm of Rasch measurement. To accomplish the tasks 2, 4 and 6, we used the descriptive analysis, multilevel regression modelling, and t-test. The data on graduates’ employment status were gathered with reliance on the methodology of longitudinal studies (task 5).
Empirical base of research
To compare the student learning outcomes (and to ensure the validity and cross-national comparability of the measures across Russia, China and India; tasks 1 and 2) we used Russian and Chinese data gathered in 2015 on academic achievements in the basic disciplines (cohorts of 1st graders and 3rd graders), as well as data on learning outcomes of Indian students collected in 2018. All the samples are clustered randomized (clusters are universities) and nationally representative. To analyze the learning progress, we used Russian data on student learning outcomes in mathematics gathered in 2015 (at the 1st grade) and in 2017 (at the 2nd grade of the same cohort). The same data were used to analyze the factors of learning progress in mathematics among Russian students (task 4). In 2018, we collected the data on employment status of Russian universities graduates (task 5) and analyzed the relationship between their employment status and the characteristics of their educational experience at university (task 6) measured at their 4th grade (in 2016).
Results of research
The analysis of validity and cross-national comparability of student learning outcomes measures across Russia, China and India (task 1) showed that the collected data can be legitimately used to compare the learning outcomes of engineering students across these countries.
A comparison of the student learning outcomes in the basic disciplines (physics and mathematics, task 2) showed that Russian students demonstrate a lower level of learning outcomes and the smallest progress compared to students from other countries. Though Chinese students show negative progress, they are the most competent professionally in comparison with both Indian and Russian students. At the same time, although Indian students show significant positive progress in mathematics, they do not advance in physics.
A comparison of the results of a special Item Response Theory model (task 3) and the strategy of vertical equating we applied in this study to analyze the learning progress showed that at the group level there is no difference between these approaches.
The analysis of institutional- and individual-level factors related to the progress of Russian students in mathematics (task 4) showed that neither the teaching styles in mathematics, nor the frequency of interactions with math instructors, nor the amount of time spent by students on assignments and attendance, nor the scores of Unified State Exams are related to the progress of Russian students. The only influential factors are the socio-economic status of the students and the type of university they study in (elite or non-elite).
In 2018, we collected the data on the characteristics of employment of graduates (task 5). This data was used to clarify the relationship between their employment status and their educational experience at university (task 6). The analysis results showed that the characteristics of educational experience at university are significantly related to the decision to continue graduate or postgraduate studies but are unlikely to determine the employment status and wage rate of graduates.
Level of implementation, recommendations on implementation or outcomes of the implementation of the results
The methodological solutions obtained in this study can be scaled to other projects and research areas. The obtained empirical results can be used to improve the functioning of the higher education system.