Applying Transformer Neural Network in Some Problems of Computer Algebra

In 2019, Guillaume Lample and Francois Charton proposed the original idea of ​​using neural networks in problems of symbolic integration and solving differential equations. In this paper, this approach applied to other problems of computer algebra: calculating the limits and the problem of choosing the optimal sequence of steps by computer algebra systems. At the first step, for the task of calculating the limits of some elementary functions, we construct an original data set consisting of pairs of lines of the example-response type. After that, we successfully train the Transformer neural network for this task. At the second step, using the example of the limit calculation problem, we construct effective (in terms of a binary classifier) ​​vector representations of successive steps of the canonical solution. Our results indicate that the vector representations found can be used to optimize computer algebra systems, in particular, using reinforcement learning methods.