The following tasks were realized in order to reach the main goal: 1) preparing and conducting the second wave of assessment (testing and gathering the context data) of first-graders’ achievements on a large-scale sample in the end of their first school year; 2) psychometric analysis of item characteristics in comparison with the international database of iPIPS project; 3) creating the final version of the computer adaptive testing (CAT) algorithm, its piloting and using it in the first cycle of CAT-based assessments. 4) conducting the comparative analysis of school readiness and school progress among first-year pupils in Russia and other participating countries; 5) conducting the statistical analysis of relationships between the context factors and school achievements of children in the end of the first year of primary school;
The assessment was based on the contemporary psychometrics principles of educational assessment. Its main tool, namely, the iPIPS instrument, is intended to provide the comparability of test results obtained on different samples at different time and with the use of partly differing test items. All test models of iPIPS are IRT (Item Response Theory) based.
The methodology of research is a combination of quantitative methods of primary and secondary data analysis. The primary analysis is IRT based including the evaluation of the test reliability (iPIPS instrument), establishing the psychometric qualities of the test items, the level of agreement with the chosen testing model (which is one-parameter Rasch model), and the analysis of scale dimensionality and DIF-analysis of the test items in relation to different groups of children. The qualitative part of research consisted of the methods of statistical data analysis, including correlation analysis, dispersion analysis, regression analysis, cluster analysis and factor analysis.
Empirical base of research
In the years 2014–2015 two cycles of iPIPS assessment were carried out (the baseline assessment in autumn and the progress assessment in spring) in three Russian regions: Moscow, Tatarstan Republic and Krasnoyarsk.
The assessment data collected from the first-year schoolchildren in Tatarstan Republic represent the empirical base of this study. In the end of the first assessment cycle, 1568 first-graders were evaluated and their data were collected. The drop-out rate of the second assessment cycle in the end of the first school year was about 17% due to the absence of children on the day of assessment. 1305 children from schools of different types participated in both assessment cycles. Test data included the results of cognitive tests in the beginning and in the end of the first year of schooling (evaluating basic cognitive skills in reading, math, and phonics), the assessment results of children’s psychological, social and emotional development, as well as detailed context information about teachers and parents.
Moreover, an international comparative research was carried out using the data from Krasnoyarsk sample and data of iPIPS assessments in Scotland.
In the end of the first cycle, baseline data from 1438 first-year pupils were collected. About 4 per cent of this sample dropped out of the project during the second assessment cycle. There were 1392 schoolchildren who participated in both assessment cycles. The sample in Scotland consisted of 6627 children in the beginning and in the end of their first school year.
Results of research
According to the project goals, in the year 2015 the second cycle of assessment was prepared, organized and carried out in order to measure the progress of large-scale sample of children who had been evaluated in autumn 2014.
As a result of the second cycle, the following databases were created (for each of the participating regions): 1) database of learning achievements of the children in the end of their first school year; 2) database of social and emotional development of the children; 3) database of behavioral characteristics of the children; 4) database of teacher questionnaires.
The results of the second assessment cycle were prepared for work and analyzed, providing such results as: 1) descriptive statistics of children’s cognitive data in the cognitive part of iPIPS were defined; 2) preliminary analysis of psychometric characteristics of the instrument was conducted; 3) the scales for math and verbal parts of the test were developed after the second piloting; 4) the scaling procedure was developed and the progress of children during the first year was assessed. The progress of children was researched in relation to various factors, including the level of children’s social and emotional development, their behaviors, family characteristics and educational environment. We managed to detect sustainable patterns of development which are specific for Russian first-year pupils (using one region as an example) in the beginning of schooling. The classification of children into groups according to their cognitive and non-cognitive characteristics seems to be the most valuable finding of this study, because in this case the assessment takes into account various aspects and it’s possible to register the direction of children’s development.
Apart from the empirical conclusion based on a large-sample findings (such as the possibility to confirm the existence of certain development types which combine social, emotional, individual and cognitive aspects of children’s development), the results of the project are valuable for teaching practices. The data obtained during the assessment might be useful for teachers in the task of building individual interactions with children in the class.
The creation of the final version of CAT, which theoretical design had been started in 2014, was completed. In the process of this work, the bank of items was created and calibrated, item characteristics were measured and corrected, parameters of CAT administration were developed (including start rules, item selection rules, item and children evaluation rules, end-of-session rules). Two versions of CAT algorithm (offline and online) were piloted in three regions (Moscow, Krasnoyarsk, Nizhniy Novgorod) on restricted samples of first-year schoolchildren.
The primary analysis of pilot data was conducted, that is, the psychometric analysis of test results including the estimation of item functioning for new and existing items, their parameter estimation according to children’s parameters, analysis of item hierarchy. As a result, the two variants of the test (booklet and computer forms) were proved to be compatible.
In 2015, the study that had been started in 2014 on a small sample of Russian children in order to compare children from different countries in the beginning of their primary school and regarding their progress in the end of the first year, was continued.
Level of implementation, recommendations on implementation of outcomes of the implementation of the results
The results of iPIPS obtained during the baseline diagnostics and the end-of-year progress assessment are intended primarily for schools and teachers of classes from the selected samples. Every school principle and every teacher receives separate reports with the results of the assessment. The reports contain the results of the school in comparison with other schools of the region, plus individual results of all participating children regarding their cognitive and non-cognitive development. The reports include recommendations on the ways to improve the educational process based on the results of the assessment.