Transformer with Adaptive Attention for Source Code Modeling

Since its introduction, Transformer has established a new level of quality in various natural language processing tasks. It is designed to work with sequential input since natural language is usually well represented as a sequence of tokens. When tackling source code related tasks the model needs to work with programming language and it is more important to utilize internal tree structure of programs. In this work we investigate the ways to integrate structural information into Transformer components. We integrate two different types of adaptive mechanism into the self-attention layer so to consider dependencies in input more accurately. The first of them, sequential adaptive attention, leads to a better performance. The second, tree-based adaptive attention, works slightly worse but uses a significantly smaller fraction of tokens in the input sequence.