I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done.

Was it initially trained on Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.


BERT is trained on both task at the same time.

  • MLM is done using the token representations.
  • NSP is done using the CLS representation.
  • $\begingroup$ Thank you! As far as I underststand token representation in multi class and Sentence prediction is binary. How are they represented in cost function? Are these two tasks concatenated? $\endgroup$ Nov 8 '19 at 14:15
  • $\begingroup$ You understood right :) To run the 2 tasks together they compute the loss of each task and sum it. You can see it here in the official source code. $\endgroup$
    – Astariul
    Nov 10 '19 at 23:30
  • $\begingroup$ Great solution to look at source code! $\endgroup$ Nov 11 '19 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.