Anonymization of Sensitive Information using Machine Learning Algorithms in Big Data Systems

Most companies nowadays desire to have intelligent systems with Machine Learning and Artificial Intelligent algorithms to improve their business operations and reduce expenses. However, it is hard and often impossible for the company to provide data with sensitive customers’ information to another outsourcing company or even to another department of the same company. In most cases, security department will not approve data transfer or data sharing. This project documents development and implementation of a solution for data anonymization before exporting datasets from the original storage. The research part of the project aims to identify a suitable Machine Learning algorithm in the named entity recognition domain for detection of sensitive information in semi-structured and unstructured data. The developed solution uses Artificial Neural Network based on Bidirectional LSTM units in conjunction with Conditional Random Fields classification algorithm for detection of sensitive information in semi-structured and unstructured datasets. The final solution utilizes Apache Spark and TensorFlow frameworks for scalable distributed computing to satisfy needs of organizations of any size and integration with standard big data processing and ETL pipelines. Anonymized dataset keeps patterns and dependencies of the original version to be suitable for future analysis and investigation by Data Scientists, without leaking customer sensitive information.