Aggregation of Noisy Textual Answers

In many practical areas, such as automatic speech or text recognition in an image, building of modern machine learning based systems require a large amount of labeled data for training and testing purposes. Often, developers have sufficient number of examples of inputs of system, but no correct answers for these examples. If a person easily interprets the objects, such datasets can be effectively labeled with crowdsourcing, but in difficult tasks, individual answers, collected with crowdsourcing, can be inaccurate and so inapplicable for use, so aggregation of multiple answers into one with better quality is required. This paper describes an approach to aggregation of noisy text answers. Approach is based on the aggregation system ``recognizer output voting error reduction'' (ROVER) and allows reach the quality close to conventional approaches with less cost to pay labor in crowdsourcing on the considered in this paper data. Developed system also gives opportunity to increase or decrease the quality of the labels with corresponding change in share of labeled objects.