Lifelong Learning for NLP Tasks

State-of-the-art systems for many NLP tasks rely on neural network models. Usually, they learn a task in two steps. First, they require training a language model on a corpus of unlabeled texts. Then, the model needs finetuning on a labeled dataset to solve the task. In this work, I will compare two scenarios for training a model that solves a task of uncertain word detection in biomedical papers. First, I fine-tune BioBERT and SciBERT language models that were initially trained on a set of biomedical papers. Second, I will fine-tune the regular BERT language model on the Wikipedia dataset of uncertain words. Then it will be domain-adapted on the dataset of biomedical papers. I show that while there is a performance gain in the score of a domain-specific model (BioBERT, SciBERT), it is not always statistically significant.