In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I correctly understood the meaning of the word "different" here. different means a different dataset or a different prediction task?
For example if we pre-train the BERT on a "sentence-classification-task" with a big dataset. Then, should I fine-tune it again on the same "sentence-classification-task" task on a smaller and task-specific data-set or I can use the trained model for some other tasks such as "sentence-tagging"?