Can I fine-tune the BERT on a dissimilar/unrelated task?

Question

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:

"During pre-training, the model is trained on unlabeled data over different pre-training tasks."

I am not sure if I correctly understood the meaning of the word "different" here. different means a different dataset or a different prediction task?

For example if we pre-train the BERT on a "sentence-classification-task" with a big dataset. Then, should I fine-tune it again on the same "sentence-classification-task" task on a smaller and task-specific data-set or I can use the trained model for some other tasks such as "sentence-tagging"?

Answer 1

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP). There were no other classification/tagging labels present in the data, as the MLM predicts the text itself and the NSP label is derived from the textual data itself. Both tasks were trained simultaneously from a single textual dataset that was prepared to feed the input text and the expected outputs for both tasks.

Therefore "different" here refers to the two pre-training tasks I mentioned: MLM and NSP.

When fine-tuning, you do not need to train again on the same sentence classification task, you just simply train it on the task you need. It is perfectly fine to fine-tune BERT on a sentence tagging task on your own dataset.