I am having trouble thinking about the token embeddings from masked attention compared to BERT.

Let's say we have 5 tokens. The embedding of the first token will be used to predict the second token, but we already know what the second token is. If we have only one decoder, then we can just use embedding of the 5th token to predict the next one (the other embeddings can be ignored). However, if we have a second decoder, then the embedding of the first token will be used by that second decoder (the intermediate embeddings from the first decoder are used).

After the first token, there are many possible next tokens. It seems to me that the embeddings for the first tokens won't be very informative (they have very little context). Why would the decoders after the first one pay attention to the intermediate results used to predict the previous tokens (1st, 2nd, 3rd, 4th) when they essentially have access to the final result (5th token)?

Can we think of the embedding of the 5th token as a sentence embedding? Can we do (5th token embedding - 4th token embedding) to obtain something similar to BERT and do token classification with GPTs?

(When I say 1st token, I was thinking of a word. I wonder if the embedding of the <START_TOKEN> is changed throughout the decoder blocks)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.