0
$\begingroup$

In BERT, NSP (Next Sentence Prediction) is for predicting next sentence based on context and Text prediction task is also for predicting next word or phrases.

So, both are for predicting next sentence or word/ phrase only and both are BERT NLP tasks, then why these two?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

NSP is not for predicting the next sentence but a binary classification task to predict whether the second sentence was originally immediately after the first or not:

During BERT's training, it is given two concatenated text sentences as input. Some tokens (15%) of the sentences are masked (the original tokens are replaced by a special [MASK] token). BERT's training objective has two parts:

  1. To guess the original tokens at the masked positions. This objective is called "masked language modelling".

  2. To determine if the second segment was following the first segment in the original document it was extracted from (i.e. true/false). This training objective is the "Next Sentence Prediction (NSP)" task.

The outputs of the masked language model task are at the positions of the masked tokens, while the output at the first position (i.e. the artificial [CLS] token added at the beginning of the text) is used for the NSP task:

enter image description here

$\endgroup$
4
  • $\begingroup$ can you please give more insights about it @noe $\endgroup$
    – tovijayak
    Jun 19, 2023 at 16:36
  • $\begingroup$ Sure, but what specific aspects are not clear? $\endgroup$
    – noe
    Jun 19, 2023 at 16:39
  • $\begingroup$ I added some clarifications. Please let me know if you have any further doubts about it. $\endgroup$
    – noe
    Jun 27, 2023 at 11:20
  • $\begingroup$ @tovijayak can you please confirm whether the answer clarifies your doubts or, alternatively, describe what aspects are not yet clear? $\endgroup$
    – noe
    Jul 27, 2023 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.