New answers tagged

0 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Multiplying Weights by √dmodel In the embedding layers, the weights are multiplied by the square root of dmodel. This is done to scale the weights appropriately and ensure that the dot product between ...
dreamg's user avatar
  • 1

Top 50 recent answers are included