Special Topics Detection in Short Natural Languages Texts

The rapid development of information technologies has led to a significant increase in the number of text documents stored in electronic form. At current rates of data growth neither single expert nor group of specialists can not cope with text processing problems. This is what determines the relevance of the development of techniques and approaches to solving problems of automatic processing, storage, and data collection. This paper discusses methods of automatic language recognition of short text messages and identify specific topics that meet the various violations of the law. The aim of the work is the analysis of search engine queries to determine the language and detect various violations. The work consists of five chapters: introduction, description of the methods of automatic language detection of short texts, description of the approach to identify the topics on the basis of specialized dictionaries, analysis of search queries using the described methods and results of the processing of search engine queries. During the preparing of this paper it was analyzed over one billion search engine queries, taken during the period from 1 to 11 April 2016. Queries were groupped by regions and filtered out 100 most active ones. The main result is the fact that an overwhelming number of queries asks the users in Russian (in different regions ranged from 47% to 53%). Besides the distribution of queries languages does no correlate with the ethnic composition of the region. The total distribution of violations persists in all regions: the most common violation is related to drugs, further violations relating to extremism and nationalism.