-2
$\begingroup$

i have i a big trouble. I don't understand transformers. I understand embedding, rnn's, GAN's, even Attention. But i don't understand transformers. Approximately 2 months ago i decided to avoid usage of transformers, because i found them hard. But i can't anymore avoid transformers. Please, help me. I want to use and understand work of transformers. How can i start to work with them?Past the fact that i want to understand their idea in general, i also want to can write/implement them using keras/tensorflow Of course i tied to read some tutorials. But i don't understand them anyway.

$\endgroup$
13
  • $\begingroup$ Transformers use many building blocks, like self-attention, layer normalization, residual connections, etc. What exactly don't you understand about transformers? Tutorials like The illustrated transformer probably do a better job explaining the whole model than we can do here, but we can try to help you understand specific aspects of it if you identify those parts. $\endgroup$
    – noe
    Feb 2, 2021 at 13:58
  • $\begingroup$ @noe, thank u. I'll read this article and say what i don't understand $\endgroup$
    – Nikto
    Feb 2, 2021 at 14:10
  • $\begingroup$ @noe, Does softmax label on photo mean how much the word relate to the first one? pasteboard.co/JMvN4Le.png $\endgroup$
    – Nikto
    Feb 2, 2021 at 16:05
  • $\begingroup$ Softmax refers to the softmax function $\endgroup$
    – noe
    Feb 2, 2021 at 16:24
  • $\begingroup$ The role of the softmax is to normalize the sum up to 1. In the example, you can see that the softmax-normalized values are 0.88 and 0.12, which add up to 1. The result of the softmax is then used as weights for the values, which are then added together. $\endgroup$
    – noe
    Feb 3, 2021 at 9:09

1 Answer 1

1
$\begingroup$

These are the answers to the specific doubts that you pointed out in the comments:

  • Transformers use many building blocks, like self-attention, layer normalization, residual connections, etc. Tutorials like The illustrated transformer are very useful to understand these blocks and how they fit together.

  • The role of the softmax is to normalize the sum up to 1. In the example, you can see that the softmax-normalized values are 0.88 and 0.12, which add up to 1. The result of the softmax is then used as weights for the values, which are then added together.

  • The decoder is very similar to the encoder, especially at training time. The main differences are that the queries are taken from the target side while keys and values are from the source side and that the attention is masked to avoid the prediction for time t to be dependent on the tokens at the same and future positions.

  • The decoder receives both the output of the encoder and the target sequence, either the full sequence at training time or the partial sequence at inference time.

  • At training time, the decoder receives the whole target sentence tokens. At inference time, we don't have the target sentence; instead, we use the model autoregressively: at each decoding step we pass as input the previous predictions, get the prediction for the next token, concatenate it with the previous step input and use it as input for the next step; at the first step of the autoregressive decoding we simply pass as input a sequence with just the special token <s>.

$\endgroup$
3
  • $\begingroup$ how to use autoregressive decoding in the code? I can't find it(at each decoding step we pass as input the previous predictions, get the prediction for the next token, concatenate it with the previous step input and use it as input for the next step;) here github.com/tensorflow/examples/blob/master/community/en/… And i also don't understand how authors form tf.data.Dataset. Sorry for my dumb questions. But i've understood how authors have formed tokenized dataset(tokenize_and_filter function) $\endgroup$
    – Nikto
    Feb 4, 2021 at 6:42
  • $\begingroup$ thank u for helping me! $\endgroup$
    – Nikto
    Feb 4, 2021 at 6:42
  • $\begingroup$ please, check it out. datascience.stackexchange.com/questions/100456/… I owe you $\endgroup$
    – Nikto
    Aug 24, 2021 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.