Fine-tuned MLM based RoBERTa not improving performance

Question

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM.

We took some sample data points (2M+) & fine-tuned RoBERTa-base (using HF-Transformer) using the Mask Language Modelling (MLM) task.

So far,

1. we did 4-5 epochs (512 sequence length, batch-size=48)
2. used cosine learning rate scheduler (2-3 cycles/epochs)
3. We used dynamin masking (masked 15% tokens)

Since the RoBERTa model is finetuned on domain-specific data, we do expect this model to perform better than the pre-trained-RoBERTa which is trained on general texts (wiki data, books, etc)

We did perform some tasks like Named Entity Recognition (NER), Text Classification, and Embedding generation to perform cosine similarity tasks. We did this on both finetuned domain-specific RoBERTa and pre-trained-RoBERTa.

Surprisingly, the results are the same (very small difference) for both models. We did try Spacy models too, but the results are the same.

Perplexity scores indicate that finetuned MLM-based RoBERTa has a minimal loss.

1. Can anyone please help us understand why MLM based model is NOT performing better?
2. Should we go for more data OR more epochs OR both, to see some effect?
3. are we doing anything wrong here? Let me know if any required details are missing. I will update

any suggestions OR any valuable links addressing these concerns would be really helpful

Huggingface discussion page: https://discuss.huggingface.co/t/fine-tuned-mlm-based-roberta-not-improving-performance/36913

Answer 1

It's great that you have fine-tuned RoBERTa on your domain-specific data and are seeing a minimal loss in perplexity scores. However, there are several reasons why the fine-tuned MLM-based RoBERTa model might not be performing better than the pre-trained RoBERTa model:

1. Data Quality: It's possible that the quality of your domain-specific data is not as good as you think. Even if you have a lot of data, if it is not representative of the domain, or if there are errors, noisy, inconsistencies or other quality issues, it could negatively affect the performance of your fine-tuned model. You may want to review your data carefully and consider cleaning, filtering or augmenting it as necessary.

2. Limited sample size: Fine-tuning a language model on a limited sample size may not be sufficient to capture the nuances and complexity of your domain-specific data. You mentioned that you used 2M+ data points for fine-tuning, but you have a much larger dataset (200M+ data points). It's possible that increasing the size of your fine-tuning dataset may lead to better performance.

3. Limited fine-tuning epochs: Fine-tuning a language model requires a sufficient number of epochs to learn the nuances of the domain-specific data. You mentioned that you only did 4-5 epochs, which may not be enough for the model to fully adapt to your data. Increasing the number of epochs may lead to better performance or using a larger batch size to improve the convergence of your model.

4. Limited fine-tuning techniques: Masked Language Modeling (MLM) is one of the fine-tuning techniques for language models. However, there are other fine-tuning techniques, such as next sentence prediction, that can be used to fine-tune language models. Also, you may want to experiment with different hyperparameters, such as learning rate, batch size, sequence length, and masking strategy to see if they affect the performance of your fine-tuned model. You could try using a learning rate schedule that increases or decreases more aggressively, or try using a larger batch size or sequence length to capture more context.

5. Limited evaluation tasks: You mentioned that you evaluated the fine-tuned model on Named Entity Recognition (NER), Text Classification, and Embedding generation tasks. It's possible that these tasks are not sensitive enough to detect the subtle differences between the pre-trained and fine-tuned models. You may want to evaluate your model on more diverse tasks, such as question-answering, summarization, or machine translation, to see if there is a performance difference between your finetuned model and the pre-trained RoBERTa model.

6. Model Complexity/Model Architecture: The RoBERTa architecture may not be the best fit for your domain-specific data and task. It's possible that the domain-specific knowledge captured by your fine-tuned model is not complex enough to make a difference in your tasks. You may want to experiment with a more complex model architecture or add more layers to your fine-tuned model or even explore custom architectures that are specifically designed for your task and domain.

Without more information about your data and task, it's hard to say exactly why the finetuned model isn't performing better. However, the explanations I've listed above are some common reasons why fine-tuning may not always lead to significant improvements in performance.

---

To address these concerns, here are some suggestions:

• Increase the size of the fine-tuning dataset and the number of epochs. You may want to try different epoch sizes and evaluate the model after each epoch to see if performance improves. Adding more data that's not representative of the domain or adding too many epochs could lead to overfitting, which would harm performance.

• Try different fine-tuning techniques, such as next sentence prediction or sequence classification.

• Use domain-specific evaluation metrics for evaluating your model. Using standard evaluation metrics like accuracy or F1 score, they may not be sensitive enough to the nuances of your domain. You may want to consider using domain-specific evaluation metrics or creating your own evaluation metrics to better measure the performance of your model.

• Use transfer learning techniques, such as domain-adaptive fine-tuning or domain-adaptive pre-training, to improve the performance of the model on your specific domain.

• Consider using different architectures, such as BERT or XLNet, to see if they lead to better performance.

• Try training a language model from scratch on your domain-specific data to see if that improves performance.

---

In terms of additional resources, you may find the following links helpful:

• The Hugging Face Transformers documentation has detailed information on fine-tuning and hyperparameter tuning: transformers
• The ULMFiT paper discusses techniques for fine-tuning LMs on domain-specific data and may be relevant to your work.
• The Allen Institute for AI (AI2) has released a toolkit for evaluating LM performance on various tasks called the "GLUE Benchmark"
• The SuperGLUE benchmark is an extension of GLUE that aims to test the limits of NLU models and may be a useful evaluation task for your fine-tuned model.

Answer 2

If you have almost the same result, it is probably due to the data itself. Is there a lot of redundant content?

Data should have some redundancy, but not too much, otherwise there is an unbalance between weights, which reduces the learning quality.

I am proposing this idea based on my experience. If your data have plenty of similarities between documents, you should reduce them to a strict minimum. For instance, if you have 10 000 articles having almost the same content, it will alter the model training.

Furthermore, redundancies are not interesting from a classification/prediction point of view. Consequently, your training data should not only be numerous but also variate to allow a good weight balance of the vocabulary corpus (or whatever model architecture).