First, it is necessary to construct valid, reliable and cross-nationally equivalent instruments for undergraduate students.
Second, it is necessary to validate the mentioned above instruments. Also it is necessary to conduct the psychometric analysis of the tests in physics, maths and informatics.
Third, it is necessary to translate into Russian the critical thinking test provided by the Stanford University. The aim of this test is to measure the cognitive skills.
Fourth, it is necessary to conduct the first round of the International Comparative Study of the quality of engineering education across the BRIC countries. Two goals must be achieved by this. First, to measure the readiness of the first-year engineering students for studying. Second, to assess the quality of the education of the third-year engineering students.
Fifth, it is necessary to compare different factors that may influence the readiness of the first-year students for studying and the quality of the third-year students’ education.
For the construction and validation of the tests and the quantitative analysis of the results the IRT and mathematical statistics were used. IRT modelling was used for the analysis of the test items in particular. Also it was used for the analysis of the dimensionality and the reliability of the tests. The DIF analysis was conducted to analyze: a) the cross-cultural equivalence of the test items, and b) the possibility to put the two countries results on the same scale. Rush model was used to construct the tests.
Empirical base of research
By the fall 2014 the pilot approbation of the 1st and 3rd grade physics and math tests had been conducted. The sample included the two groups of the 1st and 3rd grade students studying electrical engineering or computer science. Two characteristics of the universities played an important part for the sample design: status (both elite and non-elite universities had to be represented) and location (both universities located in large and small cities had to be represented). The sample consisted of 11 Russian and 11 Chinese universities. 60 students of the 1st grade and 60 students of the 3rd grade were randomly chosen in each university. If there were less students studying electrical engineering (EE) or computer science (CS) in any university of the sample, then all the students were tested. The selected EE and CS students were randomly testes in physics or math. The final sample for the pilot study included 1797 students in Russia and 1802 students in China.
The first round of the project was conducted in the 2015th. The data was collected from the 1st and 3rd grade students of 34 randomly sampled Russian universities with programs in electrical engineering or computer science.
At the first stage, experts selected majors in Russia and China that have much in common. Two groups of majors were selected: electrical engineering (EE) and computer science (CS). In Russia, the following EE majors were selected: Power and Electrical Engineering, Electricity and Nanoelectronics, Optical Engineering, Radio Engineering, Information and Communication Technologies and Communication Systems , Laser Technology. The following CS majors were selected in Russia: Computer Science and Engineering, Information Systems and Technology, Applied Informatics, Information security, Software and Administration of Computer Systems, Mathematics and Informatics.
35 universities in Russia were sampled for the research. One university decided not to take part in the research project due to the Rector election campaign coinciding in time with the research project. 34 universities took part in the research project. In these universities there were selected two samples. One was the sample of the third-year students. The second was the sample of the first-year students. 5304 were selected for the sample. 91% — 4827 students took part in the project.
Two types of tests were given to the students. The first type tests were the academic development tests. Students wrote two academic development tests — a test in math and a test in physics or informatics. The second type tests were the cognitive development tests. Students were tested in critical thinking, quantitative literacy and creativity. All students also answered the background questionnaire.
Also all the faculty members teaching physics and math were questioned on their methods of teaching. Administrators in the universities who were engaged in the education of the students were also questioned.
China was the second country that took part in the research project. The data collected from the Chinese engineering universities would be used for the research. In China the EE and CS majors also were selected and universities with these majors were randomly sampled. In the sampled universities, 1st and 3rd year students were randomly selected and tested. In China, faculty members and administrators were questioned as well.
Results of research
In the year 2015, the first round of the research was conducted. Two problems were studied — the readiness of the 1st year students for studying in the university and the quality of the engineering education after two years of studying. Faculty staff and administrators were questioned. That made it possible to discover factors of importance for the process of studying and the quality of engineering education in Russia and China. Also the students answered the Background Questionnaire. That made it possible to discover factors of importance for the readiness for studying and the quality of engineering education in Russia and China.
Valid, cross-culturally equivalent and reliable instruments were constructed for the project. These instruments may be used to measure students’ readiness for study and the quality of engineering education. These instruments include academic development tests (math, physics and informatics) and cognitive development tests (critical thinking, quantitative literacy and creativity).
Level of implementation, recommendations on implementation or outcomes of the implementation of the results
Two groups may be interested in the project. First, it is the administration of the universities. Detailed reports containing the analysis of each university results have to be presented to each university rector. So, the heads of the universities may improve teaching in their universities. They will be able to create effective policy of teaching the 1st grades students and to improve the quality of education.
The second group interested in the project is the Russian Ministry of education. The results of the project may be of help for the policy-makers in the sphere of education. They will understand the quality of the school-graduates in Russia and the BRIC countries; the quality of the Russian engineering education in comparison with the BRIC countries. Also they will understand factors important for the engineering education in Russia and the BRIC countries. They will be able to create effective policy of improving Russian engineering education and school teaching of physics and math.