• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site
For visually-impairedUser profile (HSE staff only)SearchMenu

Speech Markers of Intolerance and Computer Instruments of Their Detection

ФИО студента: Karina Shakirova

Руководитель: Tatyana Vladimirovna Romanova

Кампус/факультет: Faculty of Humanities (Nizhny Novgorod)

Программа: Fundamental and Applied Linguistics (Bachelor)

Оценка: 10

Год защиты: 2018

The paper is dedicated to the analysis of computer tools for automatic detection of lexical markers of speech intolerance. Two approaches are considered, the first one being the dictionary-oriented approach with manual editing capability and the second one being machine learning classification. The dictionary-based program compares words from lemmatized text with reference dictionaries and, when meeting an unfamiliar word, the program either ignores it or refers to the user for manual determination. The program also includes pre-processing and lemmatization of text files, allows the user to edit dictionaries, updates dictionaries during the usage and saves the detected markers in a separate text file. Machine learning classifiers, including Naive Bayes, Logistic Regression and Random Forest, were trained on a lemmatized corpus of short texts in Russian language, annotated by sentiment. The test texts were processed at the paragraph and sentence levels with the algorithm comparing the probability of the full text and the text with all occurrences of some word deleted being classified as a negative class. The words with the probabilities difference being the greatest are considered the markers of intolerance. The algorithms were tested on 10 texts of general topic from different sources with the results being evaluated on the bases of precision, recall, and F-measure metrics. All the algorithms experienced difficulties with the classification of invective lexics, occasionalisms, misspelled words, proper names, acronyms, and specific terms. Moreover, the classifiers also identified function words as markers. The dictionary approach has shown the best results due to the manual correction. Among the classifiers, Naive Bayes was the most effective, while the classification at the paragraph level showed greater precision and F-measure than the classification at the sentence level.

Full text (added May 30, 2018)

Student Theses at HSE must be completed in accordance with the University Rules and regulations specified by each educational programme.

Summaries of all theses must be published and made freely available on the HSE website.

The full text of a thesis can be published in open access on the HSE website only if the authoring student (copyright holder) agrees, or, if the thesis was written by a team of students, if all the co-authors (copyright holders) agree. After a thesis is published on the HSE website, it obtains the status of an online publication.

Student theses are objects of copyright and their use is subject to limitations in accordance with the Russian Federation’s law on intellectual property.

In the event that a thesis is quoted or otherwise used, reference to the author’s name and the source of quotation is required.

Search all student theses