I've trained a transformer model based on the pytorch tutorial: https://pytorch.org/tutorials/beginner/transformer_tutorial.html, But I found I've difficulties to understant this model's input and output connections. According to its training code, the model's input data is:

t[0], t[1], t[2],...,t[n], and its output target value should be t[1], t[2],...,t[n], t[n+1].

input: t[0], t[1], t[2],...,t[n-1], t[n]

output:t[1], t[2], t[3],...,t[n], t[n+1].

And based on my unerstanding, t[1] depends on t[0], t[2] depends on t[1], t[0], and so on, t[n+1] depends on t[n], t[n-1], ..., t[0].

My question is, since we need cut a long tokens list into multiple segments, and input these segments into the transformer model one by one, let's assume one segment has n tokens, is there any connection exists between two segments? e.g. does any state connection exist between the t[2n+2] and t[0]? Simply to say, is the 2nd segments target value t[2n+2] decided by t[0], t[1], ..., t[n], t[n+1], t[n+2]...,t[2n+1]?


1 Answer 1


For context, we are talking here about a language modeling task, that is, predicting the next word or token.

In normal transformers, there is no “state transfer” between different segments (either within the same batch or across batches) at all.

Other Transformer variants are specifically meant to model long sequences and have a memory to transfer the past hidden states to the following batches. An example is TransformerXL.

  • $\begingroup$ Does it mean we only could use normal transformer model to capture the sequential properties in every single segment training data? $\endgroup$ Commented Apr 11, 2023 at 1:55
  • 1
    $\begingroup$ Yes, that’s it. $\endgroup$
    – noe
    Commented Apr 11, 2023 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.