2
$\begingroup$

I was trying to understand transformer architecture from "Attention is all you need" paper.

enter image description here

What is purpose of stacking $N=6$ blocks of encoder and decoder? Does higher blocks represent longer phrases and learns what longer phrases attend to? While bottommost block represent single word and its attention; something like how first layer of CNN represent pixel and deeper layers represent edges and further deeper layers represents shapes (like nose, hand etc.)?

$\endgroup$

1 Answer 1

3
$\begingroup$

Does higher blocks represent longer phrases and learns what longer phrases attend to?

No, each layer can attend to arbitrarily long sequences.

While bottommost block represent single word and its attention; something like how first layer of CNN represent pixel and deeper layers represent edges and further deeper layers represents shapes (like nose, hand etc.)?

Something like this. The layers that are closer to the token embeddings represent lower-level token relations, while deeper layers learn to represent higher-level information present in the input sequences.

You can check the studies that probe Transformer models to understand what kind of language information is best represented in each layer, e.g.:

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.