New answers tagged

0 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Multiplying Weights by √dmodel In the embedding layers, the weights are multiplied by the square root of dmodel. This is done to scale the weights appropriately and ensure that the dot product between ...
dreamg's user avatar
  • 1
1 vote

Why Transformer applies Dropout after Positional Encoding?

Normal dropout does not remove whole tokens, but individual values within the vectors. Therefore, dropout does not remove 10% of the tokens in a sequence, but 10% of the values. There is a different ...
noe's user avatar
  • 26.5k

Top 50 recent answers are included