1
$\begingroup$

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I also understand there needs to be a fixed length for each input vector e.g. the max length of all sentences input into the transformer are 50 tokens, and for all sentences shorter than that padding token vectors consisting of nothing but zeroes fill the space where the additional word vectors would be.

I get that each input vector needs to have a start token at the beginning of the input vector, as well as a stop token after the last word and before the padding vectors. The integer values corresponding to the start and stop token indexes are somewhat arbitrary, but I still don't understand what the actual values of the start and stop token embeddings should be. Should they just also be vectors of zeroes? Are these values also arbitrary?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Not expert in GPT so I might miss something, but as far as I know the concept of sentence start/end markers is the same across methods: these two markers are usually considered exactly like the other tokens, typically words. This means that like any other token they should have an index (the value is arbitrary), and this index is mapped to an embedding in the same way as other words. $\endgroup$
    – Erwan
    Apr 6 at 22:14

1 Answer 1

Reset to default
1
$\begingroup$

As commented by @Erwan, the start/end of sequence tokens are like any other token: they are part of the embedding table and they are identified by their index to that table.

The vector values of the start/end tokens in the embedding table are learned during the training of the network, like the other tokens.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So padding tokens are NOT actually supposed to vectors of zeroes? They're determined through embedding like all other tokens? $\endgroup$
    – Austin Capobianco
    Apr 17 at 14:40
  • 1
    $\begingroup$ Padding tokens are usually special because they are masked out. This implies that the value of the padding token vector is irrelevant because it is ignored during the computation of the model across all layers. Start/end of sequence tokens are not masked and therefore the value of their vectors is actually used by the model. $\endgroup$
    – noe
    Apr 17 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.