Automatic Classification of Russian-Language Questions

This work deals with automatic classification of questions in the Russian language, a natural early step in building a question answering system. Basing upon Arthur Graesser's Taxonomy of Inquiries, we developed a typology of Russian questions. A corpus of 2008 questions was manually compiled and annotated according to our typology. At the first stage, we built a regular expression-based classifier. This model showed 52.7% (micro) accuracy, and this result was used as the baseline for further work. We then tested several widely used machine-learning methods (logistic regression, support vector machines, naïve Bayes) against a regular expression baseline on a held-out test corpus annotated by an external expert, using a fine-grained class set and a coarse-grained one (23 and 14 classes, respectively). The data was represented as character bi-/trigrams and word uni-/bi-/trigrams. The best results were achieved by a SVM classifier (linear kernel) that achieved the accuracy of 65.3% (fine-grained) and 68.7% (coarse-grained). At the last stage, we modified the data by reducing the typology to 13 classes, expanding the dataset and improving the representativeness of some of the question types. The training data in a combined representation of word embeddings and binary regular expression-based features was used for supervised learning. We tested a convolutional neural network against an SVM classifier with a linear kernel and questions represented as word trigram counts, as the new baseline model (60.22% accuracy on the modified data). The best result of 72.38% accuracy (micro) was achieved with the CNN model.