0
$\begingroup$

In BERT, NSP (Next Sentence Prediction) is for predicting next sentence based on context and Text prediction task is also for predicting next word or phrases.

So, both are for predicting next sentence or word/ phrase only and both are BERT NLP tasks, then why these two?

Improve this question
$\endgroup$
1
  • $\begingroup$ Hi @tovijayak. If you find the answers to your question useful, please consider upvoting them. Also, please consider accepting one (with the tick mark ✓ next to it) if you consider it correct or, alternatively, please describe in a comment why you consider it incorrect or not clear enough. $\endgroup$
    – noe
    Jun 18 at 22:57

1 Answer 1

Reset to default
1
$\begingroup$

NSP is not for predicting the next sentence but a binary classification task to predict whether the second sentence was originally immediately after the first or not:

During BERT's training, it is given two concatenated text sentences as input. Some tokens (15%) of the sentences are masked (the original tokens are replaced by a special [MASK] token). BERT's training objective has two parts:

  1. To guess the original tokens at the masked positions. This objective is called "masked language modelling".

  2. To determine if the second segment was following the first segment in the original document it was extracted from (i.e. true/false). This training objective is the "Next Sentence Prediction (NSP)" task.

The outputs of the masked language model task are at the positions of the masked tokens, while the output at the first position (i.e. the artificial [CLS] token added at the beginning of the text) is used for the NSP task:

enter image description here

Improve this answer
$\endgroup$
4
  • $\begingroup$ can you please give more insights about it @noe $\endgroup$
    – tovijayak
    Jun 19 at 16:36
  • $\begingroup$ Sure, but what specific aspects are not clear? $\endgroup$
    – noe
    Jun 19 at 16:39
  • $\begingroup$ I added some clarifications. Please let me know if you have any further doubts about it. $\endgroup$
    – noe
    Jun 27 at 11:20
  • $\begingroup$ @tovijayak can you please confirm whether the answer clarifies your doubts or, alternatively, describe what aspects are not yet clear? $\endgroup$
    – noe
    Jul 27 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.