I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments which may contain multiple sentences and the task is to predict whether the second segment is the direct successor of the first one. RoBERTa's authors proceed to examine 3 more types of predictions - the first one is basically the same as BERT, only using two sentences insted of two segments, and you still predict whether the second sentence is the direct successor of the first one. But I can't understand what the goal is in the other 2. I will cite their explanation below:
• FULL-SENTENCES: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length is at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss.
• DOC-SENTENCES: Inputs are constructed similarly to FULL-SENTENCES, except that they may not cross document boundaries. Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULL-SENTENCES. We remove the NSP loss.
So from what I understand in these two training strategies they already sample consecutive sentences, or at least consecutive sentences from neighbouring documents, and I can't see what they are trying to predict - it can't be whether they're consecutive text blocks, because to me it seems that all of their training examples have already been sampled contiguously, thus making such a task redundant. It would be of enormous help if someone were to shed some light on the issue, thanks in advance!