I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done.
Was it initially trained on Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.