Applying of Sparse Substructures for Meta-learning of Neural Networks

Increasingly extreme requirements are presented to modern deep learning models nowadays. One of the key properties of intelligence, as such, is the ability to learn continuously. However, until recently it was believed that neural networks were not able to learn a new problem, while maintaining the ability to solve previously known tasks. The reason for this is known as catastrophic forgetting was overcome several years ago, and since then attempts have intensified to train neural network models for solving several problems simultaneously. Another recent study demonstrated the presence in dense models of neural networks of sparse substructures of less than 15\% of the original network, capable of learning and solving problems no worse than their dense parents, being much more effective in terms of computation and physical memory required. Based on the assumption of the presence of the described sparse substructures in neural networks for several tasks at the same time, the aim of this work is to develop a sequential learning algorithm with their application.