2
$\begingroup$

When I read articles about Roberta, I often read the terms "transfer learning" and "fine-tuning". Additionally, they also mention "feature extraction". What are the differences between these terms ?

If I want to train the pre-trained Roberta on my dataset but only use it to calculate contextual embeddings for each caption in the dataset by taking the last hidden state of the model, what is this process called ?

$\endgroup$

1 Answer 1

2
$\begingroup$

Fine-tuning means that you modify the model to adapt it to your data.

Feature extraction means that you don't modify the model. In the case of RoBERTa, it means that you feed your data to the model, take the hidden states and use them for other thing (e.g. to keep them as vector representation to find other similar text, or as input to another model).

Your statement "I want to train the pre-trained Roberta on my dataset but only use it to calculate contextual embeddings for each caption in the dataset by taking the last hidden state of the model" is incoherent, because you are mixing incompatible things: you cannot train RoBERTa and at the same time use it only to calculate contextual embeddings. If you train RoBERTa on your data, you are fine-tuning. If you only calculate contextual embeddings you are doing feature extraction.

While it is technically possible to both fine-tune RoBERTa with your data in a masked LM task and then use the fine-tuned model for feature extraction, it does not make much sense, because, feature extraction with BERT-like models performs worse than fine-tuning. People use feature extraction just to avoid training the model. Therefore, it is a waste to train the model on a masked LM task (the same as the original RoBERTa pretraining) instead of fine-tuning it on your own downstream task (e.g. classification).

In the article To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks, you can find a study about how BERT (predecessor of RoBERTa) behaves with fine-tuning and feature extraction on different tasks.

$\endgroup$
2
  • $\begingroup$ I thought you could fine-tune the model to adapt to the dataset and then proceed with feature extraction, as the model has already learned the features from the dataset ? $\endgroup$
    – user159173
    Commented Feb 20 at 17:07
  • $\begingroup$ What you describe is technically possible. However, feature extraction with BERT-like models performs worse than fine-tuning. People use feature extraction just to avoid training the model. Therefore, it is a waste to train the model on a masked LM task (the same as the original RoBERTa pretraining) instead of fine-tuning it on your own downstream task (e.g. classification). $\endgroup$
    – noe
    Commented Feb 20 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.