Automating the Process of Managing a Data Collection System from Open Sources

The aim of this final qualifying paper is to solve the problem of automating the process of managing a data collection system from open sources. The purpose of this final qualifying paper is to simplify the management of the data collection system, which is several heterogeneous web-robots, as well as the creation of a common infrastructure to control these robots. Web-robots that collect information on the Internet are called “web-crawlers” or simply “crawlers”; these robots are applications that are usually written in the programming languages Python or Java. Crawlers have different properties and perform similar yet different functions. One of the ways to create a common system for managing heterogeneous crawlers is containerization technology. This technology allows you to place applications in containers, which are very small copies of the operating systems on which applications will run. The infrastructure is based on Docker Swarm technology, the crawlers used in the system were written in Python and Java. This work contains 3chapters, introduction and conclusion. Number of illustration - 21. Number of sources used – 25.