1
$\begingroup$

I have been working in sequence prediction tasks (very similar to language modelling) where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach like the below for data preparation

[0,1,2,3] -> [4]
[4,5,6,7] -> [8]

or

[0,1,2,3] -> [4,5,6,7]
[4,5,6,7] -> [8,9,10,11]

I have recently seen people taking output/target as one offset shifted from X. Since the goal of language modelling is to predict next token. Why and in which situation would one use this data format?

For example a famous book D2l.ai provide example format of the data as

X:  [[27. 28. 29. 30. 31.]
 [12. 13. 14. 15. 16.]]
Y: [[28. 29. 30. 31. 32.]
 [13. 14. 15. 16. 17.]]

Isn't it predicting token 28 after tokens [27. 28. 29. 30. 31.]wheras actaly Y(output) should be [32] or [32,33,34,...] ?

How would next prediction (which actually is 32) be retrieved after token 31?

Improve this question
$\endgroup$
10
  • $\begingroup$ Kindly let me know if this question is more suited for other statckexhange platform $\endgroup$
    – A.B
    Commented May 28, 2021 at 17:35
  • 1
    $\begingroup$ it is simply a format for time-based seuqences (and not specific to language sequences). It may work better in some cases, given a series of predictions may better predict a single value or even the next series $\endgroup$
    – Nikos M.
    Commented May 28, 2021 at 17:43
  • $\begingroup$ One may select a single value from the output series in case the output is a series (eg 1st or last one) $\endgroup$
    – Nikos M.
    Commented May 28, 2021 at 17:44
  • $\begingroup$ Nikos, Thankyou for the comment. Your second comment (taking last index from predcition) makes sense. Can you please elaborate more on first comment ( "given a series of predictions may better predict a single value or even the next series ") $\endgroup$
    – A.B
    Commented May 28, 2021 at 17:48
  • $\begingroup$ It was found to work, I have no other theoretical explanation at this point $\endgroup$
    – Nikos M.
    Commented May 28, 2021 at 17:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.