New answers tagged transformer
0
Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm.
Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.
0
These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states.
In Equation 1 of the ...
1
Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be
("the cat is black", "die Katze ist schwarz").
In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but ...
1
To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop:
int d_model = 512, max_len = 5000;
double pe[max_len][d_model];
for (int i = ...
Top 50 recent answers are included
