Discourse Probing of Transformer-Based Language Models

Transformer-based language models (such as BERT, GPT-2, RuBERT) are the most common architecture of language models. Such models can be used to create effective solutions for a broad range of natural language processing tasks, which is why they are widely used in natural language processing. However, the reasons of their effectiveness are not clear yet. This is why an important field of natural language processing is the research of internal mechanisms of neural language models. In particular, studies in this field are devoted to experiments aiming to prove or disprove that linguistic models have certain linguistic knowledge and to link this knowledge to certain elements of the model architecture. In this thesis, we look into the knowledge of linguistic models associated with the discursive level of language. For this purpose, we offer several experiments with Russian and English texts extracted from discourse-annotated corpora (RST Discourse Treebank for English and RuRS Treebank for Russian). With these experiments, we show that the language models BERT, GPT-2, and RuBERT are aware of discourse structures. We also show how this discourse awareness is distributed across the layers of the neural networks and analyze the features of discourse competences in each of the three models.