I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I also understand there needs to be a fixed length for each input vector e.g. the max length of all sentences input into the transformer are 50 tokens, and for all sentences shorter than that padding token vectors consisting of nothing but zeroes fill the space where the additional word vectors would be.
I get that each input vector needs to have a start token at the beginning of the input vector, as well as a stop token after the last word and before the padding vectors. The integer values corresponding to the start and stop token indexes are somewhat arbitrary, but I still don't understand what the actual values of the start and stop token embeddings should be. Should they just also be vectors of zeroes? Are these values also arbitrary?