0
$\begingroup$

I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done.

Was it initially trained on Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

BERT is trained on both task at the same time.

  • MLM is done using the token representations.
  • NSP is done using the CLS representation.
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you! As far as I underststand token representation in multi class and Sentence prediction is binary. How are they represented in cost function? Are these two tasks concatenated? $\endgroup$
    – user1700890
    Commented Nov 8, 2019 at 14:15
  • $\begingroup$ You understood right :) To run the 2 tasks together they compute the loss of each task and sum it. You can see it here in the official source code. $\endgroup$
    – Astariul
    Commented Nov 10, 2019 at 23:30
  • $\begingroup$ Great solution to look at source code! $\endgroup$
    – user1700890
    Commented Nov 11, 2019 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.