Multimodal Emotion Recognition

This thesis addresses the development of a multimodal emotion recognition system based on creating a unified embedding space aligned with Valence–Arousal–Dominance (VAD) coordinates. The proposed architecture integrates multiple modalities, including video and human pose data, into a shared feature representation. A central innovation of the system is its robustness to missing modalities, achieved through contrastive learning and embedding alignment within the VAD space. The study reviews existing approaches to multimodal emotion recognition, investigates feature extraction techniques utilizing transformer-based architectures and graph neural networks, and explores contrastive learning methods for modality alignment. It also examines the process of converting categorical labels of emotional gestures into continuous VAD representations, facilitated by advanced language models for data annotation. The proposed architecture demonstrates substantial flexibility and reliability in handling multimodal data, including scenarios involving incomplete inputs. The thesis targets professionals in machine learning, artificial intelligence, and affective computing. The thesis consists of 46 pages, 4 chapters, 7 tables, and 7 figures. A total of 38 references are cited. Keywords: multimodal emotion recognition, neural networks, VAD space, contrastive learning, transformers, graph neural networks.