Adversarial Attacks on Multimodal Language Models

Recently, multimodal tasks have become increasingly popular. This tasks are called multimodal because the model process two different types of input data. Such tasks are, for example, Image captioning and VQA tasks. Until recently, CNN+RNN models were able to handle such tasks. However, better results were shown by multimodal systems using the transformer architecture. Despite their popularity, no significant research has yet been conducted on adversarial attacks on these models. It is these attacks that reflect the vulnerability of models to minor transformations of input data. The purpose of this study is to apply adversarial attacks to multimodal models-tranformers. The attack model is LXMERT, which solves the VQA problem. The results show that even a simple type of adversarial attack successfully obfuscated the model.