Goal of research: to compare learning outcomes of engineering students in Russia and China, and to investigate individual- and institutional-level factors related to students’ educational achievements in the participating countries.
Methodology: testing, survey, and mathematical and statistical processing of the results with applications of Item Response Theory methodology (particularly, Rasch modeling) and regression analysis with cluster correction.
Empirical base of research: randomized clustered representative (for the countries) sample of engineering students. Sample included 34 Russian and 36 Chinese universities (both elite and non-elite). During the first wave of data gathering (November-December of 2015), information about approximately 2000 Russian 3rd year students and 4000 Chinese 3rd year students was collected. In November-December of 2016, the second wave of the survey for these students cohort was conducted, when the cohort was in the middle of the 4th year of study. During the second wave of the survey, higher order thinking skills (Relational Reasoning and Critical Thinking) and professional competences (in the areas of Electrical Engineering and Computer Science) were assessed as well as various individual- and institutional-level factors related to the learning outcomes.
Results of research: Evidence regarding the validity of Critical Thinking and Relational Reasoning tests as well as the validity of General Electrical Engineering test was provided. During the evaluation of psychometric properties of the tests, both Classical Test Theory and Item Response Theory were applied. We found the evidence regarding the investigated types of validity (Content and Construct validity) sufficient, what enables us to use the instruments for individual-level skills assessment.
Analysis of Differential Item Functioning (referring to the item-level comparability of results) in Critical Thinking and General Electrical Engineering professional competences tests proved cross-national comparability of results. This allows us to use the data for cross-national large-scale comparison of the Russian and Chinese students’ educational achievements.
During the cross-national comparison of the 4th year Russian and Chinese students’ results, the following conclusions were made:
- The Chinese students outperform the Russian students in professional competences in Electrical Engineering, particularly in “Basic Circuit and Its Laws”, “Basics of Digital Circuits” and most of all in “Circuit Analysis”.
- The Russian students are significantly better than Chinese students at Critical Thinking.
- The Russian students from selective universities demonstrated higher results in Computer Science, as compared to the students from non-selective universities. Particularly, they are better at “Programming and Software Engineering” and “Discrete Structures and Algorithms”. Also, they are better at General Electrical Engineering, Critical Thinking and Relational Reasoning.
- Compared to the students from regular universities, the students from National Research Universities and Federal Universities demonstrate higher achievements in all content areas of Computer Science test (that is, “Programming and Software Engineering”, “Discrete Structures and Algorithms” and “Systems”) and slightly higher achievements in Critical Thinking.
- The students from universities located in Moscow and St. Petersburg demonstrate significantly higher results in all content areas of Computer Science test.
For the purpose of investigating the factors related to the 4th year Russian and Chinese students’ learning outcomes, we used linear regression analysis with cluster correction. This allows us to obtain more reliable results in the case of clustered sample. For the analysis we used the educational achievements, obtained from the second data gathering wave (fall of 2016), and the context survey data, obtained from the first (fall of 2015) and the second data gathering wave (fall of 2016).
The results of these analyses suggest several conclusions regarding the relations between individual- and institutional-level factors and the students’ educational achievements in Russia and China:
- Socio-Economical Status of family is an important predictor of the educational achievements for Chinese students, while it does not contribute to the educational achievements in Russia.
- Both n Russia and China, the 4th year students’ learning outcomes depend on the knowledge level at the moment of the university enrollment. More prepared students achieve more by the end of the study course.
- Both in Russia and China, more frequent interaction of students with teacher related to the lower level of learning outcomes.
- Teacher practices of group activities are ineffective for the professional development of Electrical Engineering students.
- Active teaching style relates to the higher level of all investigated areas of skills in China, while it relates only to Critical Thinking in Russia.
- Honest learning approach is a significant factor of educational achievements in China, while properties of learning activities in Russia do not affect any areas of the cognitive skills.
- More than a half of Chinese and Russian students are not satisfied with their choice of major, what relates differently with their learning outcomes. In China, students satisfied with their major choice have higher levels of the all assessed cognitive skills . While in Russia, by contrast, satisfaction with the major is negatively related to the educational achievements.
- Institutional-level factors do not affect higher order thinking skills, however they affect professional competencies in Russia under control of individual -level variables. higher results in Computer Science are demonstrated by the students from Moscow and St. Petersburg, while higher results in Electrical Engineering are demonstrated by the students from National Research Universities and Federal Universities.
Moreover, in spring 2017 in Russia and China, we conducted the second wave of data gathering in the cohort of the students who took part in the project in fall 2015, when they were the first-year students. In 2017, they were finishing their second year. During this wave of data gathering, the students were administrated with academic tests (mathematics and physics) and higher order thinking skills tests (Relational Reasoning and Critical Thinking). The percent of the students who took part in the project for the second time is 84% in Russia and 95% in China.
Level of implementation, recommendations on implementation or outcomes of the implementation of the results
The results for each Russian university containing the comparison with other Russian universities were sent to each participating university in the format of report. At the institutional level, the findings might be used as an independent method of education quality assessment.
At the national level the research results might be used for the elaboration of pertinent public policies aimed at improving the quality of engineering education in Russian universities.