Implementation of Automated Machine Learning for the Support of Data Quality Processes

On a case by case basis, data quality issues are often easily identifiable, monitorable and resolved. However, when an environment of ever-increasing data volumes is considered, methods are needed to manage quality issues with the purpose of making sizeable data quantities manageable and useable. To address the need for a solution, the present state of the art in data quality management and automated machine learning was examined and discussed. On the basis of a medium-sized dataset with 5000 rows, an automated machine learning binary classification pipeline was implemented, evaluated, and validated through the use of prototyping and k-fold validation. The firstly trained model had an initial MCC score of 0.697 on the training dataset compared to the score of 0.949 for the best-scoring machine learning model of an Ensemble. Furthermore, the Ensemble model attained a 98.8% accuracy on the holdout dataset. Finally, an implementation of the Ensemble model into different data quality processes was considered and modelled.