Transformer-Based Architectures for Tabular Data

Currently, the methods of deep learning show state-of-the-art quality for a wide range of tasks. Over the years of research, various task-specific or even field-specific models have emerged, i.e., recurrent neural networks or Transformers for natural language processing and convolutional neural networks for computer vision tasks. One of the fields that lack such specialized architectures is tabular data processing, for which fully-connected neural networks are still in use, and the standard methods are non-differentiable models, e.g., gradient boosting decision tree. This fact, along with a prevailing tendency towards using Transformers out of the scope of natural language processing, raises the question of the applicability of Transformers for tabular data. This work studies this possibility and the use of Transformer-specific architecture modifications to force a proper inductive bias. We propose T-Transformer, which performs better than CatBoost, TabNet, or fully-connected networks on some datasets. T-Transformer does not require many parameters, yet resulting in competitive quality. We also study the interpretability of the proposed method.