Character Level Models for Hashtag Segmentation

The main purpose of this work is to investigate the problem of hashtag segmentation in Russian. We treat a hashtag as a sequence of Cyrillic letters, which form a few words with omitted white spaces. Although the majority of existing methods use unsupervised language models, we treat this problem in a supervised way by using a neural-based character level models. I provide a brief overview of different approaches which were used recently by several research groups and propose an application of LSTM recurrent neural networks to this problem. I assume that RNNs as a baseline NLP technique are very suitable for solving a problem of segmentation of multiple words joined together. This technique is being tested on real-life data. The data was collected from an open source – social media VK. Usually, a post in this social network can be divided into the text and the corresponding hashtags. As a result of this division, a collection of texts in Russian and a list of hashtags were created. The training data is a synthetic data generated from the collected corpus of texts. And the test data is a collection of widely used real hashtags. The dictionary approach, described in the chapter "Natural Language Corpus Data" by Peter Norvig from the book "Beautiful Data", was taken as a baseline method in the problem of hashtag segmentation. In particular, the probabilistic language model, which makes the decision whether to split the hashtag or not, based on the frequency of unigrams and bigrams. The experiments depict that the character level model outperformes the dictionary method. Despite the experiments with the architecture of the model, I have considered the active training approach in order to achieve a greater interpretability of the model. The main stages of work: literature review, collection of texts, methods, experiments and results, software implementation and conclusion.