Investigating Loss Functions for Effective Training of Models that Estimate Semantic Textual Proximity

The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in natural language processing (NLP). Here, we explore the idea of using a batch-softmax contrastive loss when fine-tuning large-scale pre-trained transformer models to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduce and study a number of variations in the calculation of the loss (normalization ways, symmetrization, aligning scores on the similarity matrix diagonal) as well as in the overall training procedure (data shuffling, trainable temperature and sequential pre-training). Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classification, ranking, and regression. In particular, for the Snopes task (ranking task), we implement a full pipeline using BSC loss, which achieves state-of-the-art results. The importance of each proposed modification of the formulation and training methods is demonstrated both theoretically and experimentally using an ablation study. Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of the contrastive loss in NLP.