Data Markup and Quality Control with Crowdsourcing

Modern machine learning algorithms require training on large data sets. Crowdsourcing has become a popular approach to collecting and labeling large data sets in a shorter time frame and at a lower cost compared to data collection and annotation by a limited number of experts. However, due to the fact that crowdsourcing performers are unprofessional and have different levels of knowledge, such labels are much noisier than those received by experts. For this reason, special methods such as aggregation, incremental re-labeling, and pricing need to be used to collect high-quality data within a limited budget. As part of this work, provided a part of the unique industry experience in the field of effective data labeling using crowdsourcing, the use of which is developed in such industrial companies as Yandex, Amazon, and Microsoft. Discussed the basic concepts of data markup through popular crowdsourcing sites and analyze and develop popular components for effective data acquisition. The main theoretical results in the field of efficient aggregation, incremental reassignment and dynamic pricing, their strengths and weaknesses, as well as their applicability in real-world problems will be considered.