• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site

Factors of educational quality

Priority areas of development: sociology, humanitarian

Direction 1 - Assessing and Improving the Quality of Higher Education: An International Study

Goal of research: to conduct a comparative research of preparedness of BRIC schools graduates (particularly graduates who are entering engineering majors) for university, and of the quality of their education after two years of university.

Objectives for 2014:
  1. To develop tools for assessment of students’ knowledge in Mathematics and Physics at the beginning of their education and at the third grade;
  2. The approbation of the instruments in Russia and China.
Research object: engineering students in Russia and China.

Research subject: the quality of students’ training  in mathematics and physics, as well as their level of critical thinking.


The expert-oriented approach was used for the test development. This approach allows creating a test with the existing items pools. Test items were provided by participating countries - Russia, China and the USA. All test items were translated to English and then to Russian and Chinese.
The expert evaluation consists of two stages. First, the subject areas were evaluated by experts from Russia and China. Experts were asked to rate the importance of subject areas. The list of items was created based on selected topics. Next step, these items were rated by experts. Thus we created a  set of items based on the experts’ estimates.

Empirical base of research:

A pilot study was conducted in Russia and China in October and November 2014. The study involved more than 5,000 students from 21 universities.

Results of research:

The following toolkits were developed and piloted:
  1. The general cognitive skills (physics and mathematics) and critical thinking tests for the 1 grade students;
  2. The general cognitive skills (physics and mathematics) and critical thinking tests for the 3 grade students;
  3. Contextual  questionnaires for students, teachers and administration.
The item banks for the test development process were provided  by China, Russia and USA. The critical thinking test was provided by ETS (Educational Testing Service), USA. This test was translated and adapted to the needs of the project.

Based on the results of the pilot study, next year we plan to run the main study on the bigger sample. The baseline of the 1st year students will be compared with the results of the 3rd year students, to find their mean progress during the training. There also will be conducted the analysis of factors that are associated with the students’ results and their progress during the training. The particular emphasis will be made on the international comparison between the countries.
Direction 2 - Relation of educational contexts and teaching practices to students’ outcomes

Research purpose :

This research direction consists of three parts. One part had rather methodological focus. We were interested in how different approaches to family socio-economic status operationalization relate to discrepancies in research results. Other two parts were devoted to different factors driving secondary school students’ educational outcomes and trajectories in Russia. In the second part we investigated factors of secondary school students’ self-efficacy in a subject. Finally, in the third part we focused on effect of the whole class and referent classmates on high school students’ performance and after-school educational trajectory choice.

 Methodology :

Different methods were used depending on the research questions. The first part of the research was based on deep literature review. We analysed approaches to family SES operationalization. For the second part of the study we did regular OLS regressions. For the third part we did social network analysis and series of logistic regressions.

Empirical base of research:
We used data of Russian longitudinal panel study of educational and occupational trajectories (TIMSS 2011, PISA 2012 and fall 2013 waves).

Research results:

The results of the first part of the study have shown that there is quite a wide range of SES proxies used in social science research. The most frequently used operands are parental occupation and education, indicators of family financial capital (income), cultural capital indicators, such as books in the home and other cultural possessions. Our analysis shows that question about parental occupation has the most parents-child agreement. Question about parental education, on the contrary, has low parents-child agreement and is more tended to be missed by the children. All variables need additional conversion to become appropriate for use in comparative international studies. Number of books in home is a widespread indicator of cultural capital or family educational resources. This question has quite high level of agreement and low non-response rate especially when its categories grouped in bigger ranges. Concerning practical needs of research it can be said that researchers should pay more attention to parents’ occupation rather than education when they operationalize SES.

While doing the second part of the research we found that self-efficacy in math is correlated with some out-of-school factors such as gender and family SES. Among in-school factors the most important is average class achievement in particular subject. When low performing student is in a high-performing class this decreases his or her self-esteem. The same time if teacher uses different math tasks for students with different level of abilities in math this will strengthen big-fish-in-little-pond effect for lower-performing students. 

The third part of the study was devoted to peer-effect on performance and educational trajectory choice. We found that student’s achievements in algebra are more related to the whole class average performance, rather than to the referent classmates’ outcomes. However in physics student’s achievements show statistically significant correlation with both class average scores and referent classmates average scores. High performance in physics increases a chance of STEM choice, while high performance in math does not. It could be so probably because performance in math may reflect general ability rather than interest to STEM. Proportion of classmates who are going to enter STEM department in college turned out to be not significantly related to student trajectory choice. However if some referent classmates choose STEM and these classmates are central persons in class network this increases the probability for a student to go to STEM as well. These inferences demonstrate that studying peer-effect only by estimation of average performance in class can lead to biased estimates. Therefore it is crucial to include in-class network characteristics in analysis when peer-effect is in interest.
Direction 3 - Comparative Study of Secondary School Mathematics Teachers’ Beliefs and Practices"
Research purpose is to compare beliefs of secondary school math teachers in Russia and other countries.

Methodology :

In order to form the scales, evaluate their psychometric properties, and check the cross-cultural equivalence of latent constructs, two approaches were used, Item Response Theory (IRT) and multi-group confirmatory factor analysis (CFA). The analysis was conducted using Winsteps and MPlus software. Also some statistical methods were used: correlation, ANOVA and descriptive statistics. To collect qualitative methods data for validity study the structural interview were conducted.

Empirical base of the research :

To estimate and describe teachers’ beliefs and practices the NorBA survey was conducted. Also the data from qualitative study were collected. 
The data were collected in Latvia, Estonia and Russia. The target group was secondary school math teachers (in grades 7-9). In Latvia 390 mathematics teachers from different regions participated in the survey (95 are Russian-speaking). In Estonia 332 teachers took part in the survey (92 are Russian-speaking). In Russia research was conducted in one big region in Siberia. 1096 teachers took part in the survey. It is 40% of the general population and the sample is representative relative to population of the region. 

Research results:

The validity and reliability of NorBA questionnaire were checked. Two scales emerged, measuring the level of constructivism and the level of traditionalism in teacher beliefs. Both scales are unidimensional, and all items have good psychometric characteristics and fit the model. The validity was estimated by DIF analysis, CFA, correlation with other scales and qualitative methods. All analyses showed a good quality of the instrument.

Differences between math teachers’ beliefs in different countries are statistically significant. Russian teachers are more constructivists than teachers in Latvia and Estonia. Most of teachers in Estonia compromise both approaches: their views about good teaching include construction of knowledge and accept also transmission of knowledge in combination with it. Russian teachers are strongly oriented towards systematic approach to math teaching.  It supports the claim that in Russia traditions of high quality mathematics education are still strong; emphasis on rigorous proofs, logic, exact definitions and a precise use of the mathematical language is a characteristic of Russian mathematics education. Teachers’ beliefs are realized in their classroom practice.
The research results will help to evaluate school mathematics education in Russia in comparison with other countries and to predict its development.
Direction 4 - Solving of contextual tasks: a role of subject knowledge, general abilities and teachers’ factors

Research purpose:

This research direction consists of three parts:
The first aim was to understand whether the difficulties of Russian students in solving PISA math problems are specific and caused by some particular characteristics of these problems or the difficulties are rather general, but revealed themselves because of specific, uncommon design of these problems.

The other study was dedicated to developing of a principal framework for word math problems that are intended for assessment of a competence to apply mathematics to real life situations.
And the third part was dedicated to revealing specific math teaching practices which are correlated with students’ ability to use their school knowledge in the non-academic context.


Firstly, we conducted an experiment to understand the difficulties that Russian students meet when they solve PISA math problems. Each of them was asked to do two contextual mathematical tasks. Thinking aloud protocols (both concurrent and retrospective) were used to gather data. Protocols and video records were coded and analysed.
The second part of the research was based on deep literature review. And for the third part we used regression analysis and chi-square test.

Empirical base of the research:

For the first part we conducted an experiment with math tasks and for that purpose 60 students from 8-10 grades were selected in one public secondary school in Moscow. And in order to estimate and describe math teaching practices we used data of TIMSS 2011 and PISA 2012 study in Russia. An ability to apply school knowledge in the non-academic context was considered as a student’s results in TIMSS and PISA. 

Research results:

The results of the first part suggested that partly failures in problem solving could be explained by difficulties in modeling process, that is they were specific for these particular problems. However the vast majority of mistakes reflected the poor general problem solving skills. We found that inability to visualize clearly a disposition of the key task elements has a detrimental effect on performance. Also the need to develop a solving strategy, validate their own solution made difficulties for students.  We draw conclusions for teaching mathematics and assessing a competence to use mathematics to solve problems in real life.

In the results of the other study we proposed one possible theoretical model, based the key characteristics real everyday situations where math knowledge would be applicable. We have considered the following model parameters as obligatory: 1) everyday semantic; 2) novelty, untypical for a problem solver; 3) consistency between a question and a context (a situation) of a problem; 4) moderate structuring; 5) requiring some curricular math knowledge for solving. We have shown how the proposed model parameters can be applied to distinguish between different math contextual problems and develop appropriate problems.

Finally, there was conducted analysis of math teacher practices in classes with poor and excellent PISA results as it was suggested that specific teaching practices are correlated with students’ results in PISA. At the first stage, the most and the less effective schools were identified by regression analysis with PISA and TIMSS scores on participants of longitude survey. However, further statistical analysis in those different of classes hasn’t revealed the difference in math teacher practices, and therefore it was suggested to start a series of video-observation in math classes and in-depth interviews with those math teachers.


Carnoy M., Вальдман А. И. How well do you need to know it to use it? / Высшая школа экономики. Series EDU "Education". 2014. 
Tyumeneva Y. A., Kuzmina Y., Kardanova E. IRT Analysis And Validation Of The Grit Scale: A Russian Investigation / Высшая школа экономики. Series PSY "Psychology". 2014. No. WP BRP 24/PSY/2014. 
Carnoy M., Khavenson T., Costa L., Fonseca I., Marotta L. Is Brazilian Education Improving? A Comparative Foray Using PISA and SAEB Brazil Test Scores / Высшая школа экономики. Series EDU "Education". 2014. No. 22. 
Zakharov A., Carnoy M., Loyalka P. Which teaching practices improve student performance on highstakes exams? Evidence from Russia // International Journal of Educational Development. 2014. No. 36. P. 13-21. doi
Carnoy M., Froumin I., Loyalka P. K., Tilak J. B. G. The concept of public goods, the state, and higher education finance: a view from the BRICs // Higher Education. 2014. Vol. 68. No. 3, Published online: 25 February 2014. P. 359-378. doi
Khavenson T., Solovyeva A. Relationship between the Unified State Exam and Higher Education Performance // Вопросы образования. 2014. 
Ларина Г. С., Кузьмина Ю. В. Стало ли российское образование эффективнее за восемь лет? (на данных международного исследования PISA), in: XIV Апрельская международная научная конференция по проблемам развития экономики и общества: в 4-х книгах. Книга 4. Москва : Издательский дом НИУ ВШЭ, 2014. С. 565-574.