1
$\begingroup$

I was trying to understand transformer architecture from "Attention is all you need" paper.

The paper says:

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.

What does it exactly mean by "different representation subspaces". Can you give intuitive example in terms of natural language conversation example. For example, in sentence "Jane went to Africa during summer", query matrix $Q$ correspoding to word "Africa" can comprise of different queries "Who went to Africa?", "When went to Africa?". What are "different representation subspaces" here? Or with any other example of your choice?

$\endgroup$
1
  • $\begingroup$ Please consider upvoting the answer if it was useful. Also, please consider accepting it if deemed correct, or alternatively, please indicate why you think it's not correct or not clear enough. $\endgroup$
    – noe
    Apr 27 at 20:05

1 Answer 1

1
$\begingroup$

It means that, because each attention head has its own projection, it can learn to capture different aspects of the sequences. For instance, one sentence may focus on negation, while another head may focus on coreference resolution.

While these examples serve as illustrations of the concept, they are probably not happening in an actual trained Transformer model, because they are basically bound to human interpretation of language and Transformers are probably not bound by them, especially taking into account that most Transformer models use sub-word tokens instead of word-level tokens, which limit the applicability of word-centered interpretation of language.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.