• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site

Detailed Analysis of the Results of Education Quality Research in the Context of Economic Indicators in Education

Priority areas of development: state and public administration
2013
The project has been carried out as part of the HSE Program of Fundamental Studies.

Part 1

Research object: engineering education in BRIC countries

Research purpose: to conduct a comparative research of the preparedness of BRIC school graduates for university (particularly graduates who are entering engineering majors), and the quality of their education after two years of university. The aim in 2013 year was to develop a proposal for research into the quality of engineering students preparedness in physics and mathematics in Russia and the same students in other BRICs. The main challenge for 2013 was the analysis of engineering education in BRIC countries, the quality of engineering graduates (including school and university graduates) in order to understand whether it is necessary and possible to develop one test to assess the quality of engineering graduated in BRIC, and one test to assess the quality of engineering education. Consequently, the research of 2013 year is the first step in a big research designed for 2014-2017 years.

The present research is a continuation of previous work by laboratory researchers involved in higher education (particularly, engineering) analysis in BRIC.

Empirical base of research:

1. We analysed data from more than 400 universities in Russia that provide education in engineering majors

2. 4 mathematics lecturers from 4 different Russian universities and 176 first-year students of the same universities were surveyed on the questions on mathematics from JEE Main test (India) and CEE (China) in order to understand whether they can solve such tasks

Research results:

1. Analysis of literature revealed the main characteristics of higher education in BRIC countries: differentiation of universities, large number of private universities and high role of government in higher education.

2. Methods for choosing engineering graduates differ between BRIC countries in syllabus and configuration

3. Research on USE scores showed that there are a lot of universities that take very weak students on engineering majors (average USE score is less than 55), and there are only 30 universities that take good students (average USE score is more than 70)

4. Expert (mathematics lecturers) and student assessment showed that such tasks can be solved by Russian school leavers and can be used as a base for a future test for BRIC countries.

Implementation of research results: the results of this research will be used as a basis for the development of international tests for the assessment of engineering education quality.

Part 2.1

Research object: resilient and “antiresilient” students.

Research purpose is to study factors which drive resilience and “antiresilience”.

The empirical basis of the research is  PISA and TIMSS students’ and schools’ data.

Research results: Compared to other countries, there is an average number of both resilient and “antiresiliet” students in Russia (about 3%). Practices that effect resilience are reading for pleasure, practices of control in learning used by students, practices of support used by teachers and teachers’ stimulation of reading. Finally we find that memorization used by students supports “antiresilience”.

Part 2.2

Research object: USE's predictive validity

Research purpose: To analyze the relationship between USE scores and a student’s further academic achievement in university. Through that relationship, we will examine the predictive validity of the USE as a tool for university admission.

Empirical base of the research: The sample consisted of students from five universities across different Russian regions. Four of the universities provided information about all students enrolled during 2009-2011, and one provided information only about the two largest departments within the same cohort. Overall, the sample data included 65 departments and more than 19,000 students.

Research results:

The total USE score predicts further performance accurately enough to consider this exam as a valid tool for selecting applicants. The regression models resulted in a percentage of variance explained ranging from 15 - 35% in different departments. The predictive ability of specific subject scores is approximately the same, but the examinations in Mathematics and Russian language are most likely to be the best predictors of academic performance. The association between the USE results and academic performance during the second year and higher could be observed only indirectly through the first year performance.

Implementation of the research results:

Suggested methodological approach can be used as a tool for a monitoring survey for any university.

The tool for evaluation of freshmen can be based on this methodology

Part 2.3

Title: «­­­The characteristics of the educational environment and learning outcomes in primary school (based on SAM - Student's Achievement Monitoring)»

Research object: the characteristics of the educational environment in primary school.

Research purpose: to identify and describe the characteristics of the educational environment related to academic results in primary school in the context of Vygotsky's theory.

Empirical base of the research: the results of testing students in grade 4 (in one region of the Russian Federation, 4406 students), and questionnaire for their teachers.

Research results: The results of SAM test in the region were described. The scales for constructive and traditional teachers’ practices were composed and their reliability and validity were confirmed. The characteristics of the educational environment associated with successful test results were identified and analyzed.

Implementation of the research results: The results can be taken into account by regional and federal authorities when making decisions and devising policy.

Part 2.4

Research object: shadow education effects on high school students’ academic outcomes.

Research purpose was to provide rigorous evidence about the causal impacts of participating in shadow education on college preparation.

Empirical base of the research. To estimate the impact of participating in shadow education on student achievement in Russia, we rely on data from a large-scale, representative survey. The survey was conducted in May 2010 in three Russia regions: Pskovskaya and Yaroslavskaya oblasts and Krasnoyarsky krai. The schools in the dataset were sampled using a stratified random sample design. In each sampled school, all students in the 11th grade were surveyed. Altogether, the dataset contains information on 2,938 final-year (grade 11) students in 127 schools. Besides students three more groups of respondents were surveyed within each school: Russian language and math teachers, and school principals. Finally, in the summer of 2010, after USE test results were released, each student’s individual USE scores in math and Russian language were collected.

Research results. We find that participating in shadow education positively impacts high-achieving students but not low-achieving students. Participating in shadow education further does not lead students to substitute time away from other out-of-school studies. Instead, the results suggest that low-achieving students participate in low-quality shadow education, which, in turn, contributes to inequality in college access.

Implementation of the research results: Policymakers that are concerned about inefficiency in the provision of shadow education (in helping low-achieving students) may wish to find ways of providing this information to low-achieving students.

Part 3

The project was implemented within the framework of the HSE Basic research program.

Research object: transfer of learning

Research purpose: to combine PISA and TIMSS data collected on the same sample of students to examine the link between the level of mastery of subject knowledge in math and knowledge transfer into out-of-subject context.

Empirical base of the research: Russian students (n= 4241) from 229 schools who took part in both TIMSS-2011 and PISA-2012. There were 49.8 % girls and 50.2% boys with a mean age of 15.9 years (in 2012) (SD=0.5). The sample was representative for Russian 14-15 year-old students in terms of school size and school location.

Research results:

We found a positive relation between the mastery level of a subject and the ability to transfer learnt math to out-of-subject context. The better the mastery level of mathematics, the higher probability that learnt math will be transfered to contextualized situations. This link was not linear though: only the highest mastery level contributed significantly to the transfer. The other levels only slightly differentiated performance in transferring.

Implementation of the research results:

The impact of formal subject knowledge on its transfer is discussed in the frame of theory of transfer as well as reinterpreting achievement testing.

International partners: Martin Carnoy

Part 4.1

Title: "Mathematics teachers’ professional profile "

The project was implemented within the framework of the HSE Basic research program.

Research object: mathematics teachers of ISCED2 level.

Research purpose: To identify and describe the psychological and professional features of an ISED2 mathematics teacher in Russia and compare the professional profile of a Russian math teacher with the results of cross-cultural international studies.

Empirical base of the research: To estimate the socio-demographical of teachers, we used data from the website of Federal Agency of Education. To describe professional knowledge of mathematics teachers we used the TEDS-M database. To estimate and describe school climate, teachers’ beliefs and practices we conducted a field trial of NorBA questionnaire.

The data for the field trial were collected in Latvia, Estonia and Russia. The target group was secondary school math teachers (teaching grades 7-9). In Latvia 385 mathematics teachers from different regions participated in the survey. In Estonia 327 teachers took part. In Russia research was conducted in one big region in Siberia. 1223 teachers took part in the survey. It is 40% of the general population and the sample is representative relative to the population of the region. 

Research results:

a. 98 % of teachers - women, 96 % have higher education;

b. 25% are teaching in big cities (over 200 thousands citizens), 30% - in small towns (less than 100 thousands), 29% - in villages and 14% in urban settlements; 37% of teachers have more than 26 students in a class, 15% - less than 10.

c. The average age of teachers was 48 years, while there are only 4% of teachers younger than 26 years, and 20% is over 55 years (retirement age). Thus, we can talk about the general "aging" of teachers and a small tributary of new specialists. This is also confirmed in results of the TEDS-M: only 5% of graduates from pedagogical universities are confident that they will work in school, while 73% do not consider teaching a promising career.

d. Graduates of Russian pedagogical universities, future teachers of mathematics, have a high level of both fundamental mathematics and pedagogical knowledge according to TEDS-M.

e. Differences between math teachers’ beliefs in different countries are statistically significant. Russian teachers are higher in their constructivism then teachers from Latvia and Estonia.

f. Most of teachers in Estonia compromise both approaches: their views about good teaching include construction of knowledge and accept also transmission of knowledge in combination with it.

g. Russian teachers are strongly oriented towards a systematic approach to math teaching. This supports the claim that in Russia traditions of high quality mathematics education are still strong; emphasis on rigorous proofs, logic, exact definitions and a precise use of the mathematical language is characteristic of Russian mathematics education.

h. Teachers’ beliefs are realized in their classroom practice.

 Implementation of the research results: The results will help to evaluate school mathematics education in Russia in comparison with other countries and to predict its development. 

 International partners:

1) Markku Hannula, PhD, University of Helsinki,Teacher Education, Faculty Member,http://helsinki.academia.edu/MarkkuHannula

2) Madis Lepik, PhD, Tallinn University, Institute of Mathematics and Natural Sciences ; Associate Professor, https://www.etis.ee/portaal/isikuCV.aspx?PersonVID=38069&lang=&lang=en

Part 4.2

The project was implemented within the framework of the HSE Basic research program.

The project was implemented within the framework of the HSE Basic research program.

Research object: impact of teachers’ characteristics on students’ performance in easiest and hardest mathematical tasks

 Research purpose: describe the national achievements in international assessments programs in terms of performance in easiest and hardest mathematical tasks;determine the impact of teachers’ characteristics on students’ performance in easiest and hardest mathematical tasks.  

Empirical base of the research: national sample of TIMSS-2011 (8 Grade) 5,008 students (49,2% females и 50,7% males): 211 schools, 230 classes and 231 math teachers.

Research results: Teachers’ practices can be divided into three groups: asking students to solve the hardest (advance level) tasks, asking students to solve routine tasks; asking students to do math under teacher supervision;

- practicing with hardest tasks associates with higher performance in TIMSS, practicing with routine tasks associates with lower performance; 

- the mismatch of TIMSS data to a planned type of analysis led to shifting to another method: dividing the students into six TIMSS-groups according to their performance in TIMSS; identifying 10 (and 20) the most difficult PISA items (based on Rasch Model); determining a percentage of the most difficult PISA items answered correctly in every TIMSS group. This percentage served as a measure of the ability to transfer school math knowledge to out-of-subject and apply it to an out-of-subject context. In details see Direction 3 in this report.

Implementation of the research results

The analysis conducted can be considered as a preliminary stage for further research.

International partners: Martin Carnoy

Publications:


Nezhnov P., Kardanova E., Vasilyeva M., Ludlow L. Operationalizing Levels of Academic Mastery Based on Vygotskys Theory: The Study of Mathematical Knowledge // Educational and Psychological Measurement. 2015. Vol. 75. No. 2. P. 235-259. doi
Loyalka P., Zakharov A. Does Shadow Education Help Students Prepare for College? / NRU Higher School of Economics. Series EDU "Education". 2014.
Енчикова Е. С., Карданова Е. Ю. Оценка качества образования в начальной школе (на основе инструмента SAM) // Качество образования в Евразии. 2014. № 2. С. 83-105.
Tyumeneva Y. A., Вальдман А. И., Carnoy M. How Well Do You Need to Know It to Use It? / NRU Higher School of Economics. Series EDU "Education". 2014.
Нежнов П. Г., Карданова Е. Ю., Рябинина Л. А. Исследование процесса присвоения учебного содержания // Вопросы образования. 2013. № 4. С. 168-187.
Carnoy M., Khavenson T., Ivanova A. Using TIMSS and PISA results to inform educational policy: a study of Russia and its neighbours // Compare: A Journal of Comparative and International Education. 2015. Vol. 45. No. 2. P. 248-271.