Behavior of Modern Pre-Trained Language Models Using the Example of Probing Tasks

In this paper we are focusing on language modeling, in particular, on the pre-trained language model BERT and its behavior when solving the probing task of masked language modeling. The theoretical aspects of pre-training and researching of language models were considered, and namely the theory of semantic roles, language frames, presupposition, negation from linguistic point of view. To conduct the experiments for each of these aspects, the Russian-language corpus was collected. It consists of the educational texts for Russian language learners and marked up with the help of the National Corpus of the Russian language. There are about 2,500 words, or 20,600 characters, in the corpus. To analyze subcorpuses, a script in python was written using libraries with pre-trained models and tools for working with them. The aim of the work was achieved: the behavior of the pretrained language model BERT was investigated in the probing task of masked language modeling for the Russian language and the cases, when the model provides unsatisfactory results, were linguistically described. In terms of quality metrics (precision at k and recall at k, as well as the proportion of words, semantically related to the target word by hypo-hyperonymic relations in predictions), the multilingual BERT is recognized as the best model. Because of this the hypothesis, that models trained in Russian-language material has a better prediction, was not true. However, the results of different experiments generally demontrates balance between all three models, consequently, they all have strengths in relation to certain linguistic phenomena.