Deep Learning Models for Morpheme Segmentation

This master's thesis deals with the problem of morpheme segmentation from the point of view of subword-based tokenization. Contemporary approaches to tokenization widely employ unsupervised learning algorithms to separate words into smaller units which makes it possible to significantly reduce the size of vocabulary, compared to word-based tokenization. These algorithms implicitly model a language's morphological system to a certain extent, and that proves to be immensely beneficial on downstream tasks. However, the resulting subword units only partially match with actual morphemes. Thus, linguistically motivated subword tokenization appears to be the next logical step. Such tokenization entails the usage of supervised learning, which in turn requires a considerable amount of labeled data that is relatively hard to obtain for this task. Accordingly, we propose a novel method for generation of labeled data for arbitrary word forms in Russian language using the Russian Wiktionary. In addition, we carry out experiments on several deep learning models in an extensible and reproducible environment to gauge their performance on the task of morpheme segmentation. We present the code for the data generation and the experiments, as well as the data itself in open-source repositories.