1
$\begingroup$

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:

"During pre-training, the model is trained on unlabeled data over different pre-training tasks."

I am not sure if I correctly understood the meaning of the word "different" here. different means a different dataset or a different prediction task?

For example if we pre-train the BERT on a "sentence-classification-task" with a big dataset. Then, should I fine-tune it again on the same "sentence-classification-task" task on a smaller and task-specific data-set or I can use the trained model for some other tasks such as "sentence-tagging"?

$\endgroup$
1
  • $\begingroup$ Please consider upvoting or marking as correct the answers you find useful. $\endgroup$
    – noe
    Nov 1, 2020 at 11:48

1 Answer 1

2
$\begingroup$

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP). There were no other classification/tagging labels present in the data, as the MLM predicts the text itself and the NSP label is derived from the textual data itself. Both tasks were trained simultaneously from a single textual dataset that was prepared to feed the input text and the expected outputs for both tasks.

Therefore "different" here refers to the two pre-training tasks I mentioned: MLM and NSP.

When fine-tuning, you do not need to train again on the same sentence classification task, you just simply train it on the task you need. It is perfectly fine to fine-tune BERT on a sentence tagging task on your own dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.