What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Question

As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism can be implemented in CNN?

Answer 1

Here's the list of difference that I know about attention (AT) and self-attention (SA).

1. In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - the attention looks at the inputs of the same layer where it's applied.

2. AT is often applied to transfer information from encoder to decoder. I.e. decoder neurons receive addition input (via AT) from the encoder states/activations. So in this case AT connects 2 different components - encoder and decoder. If SA is applied - it doesn't connect 2 different components, it's applied within one component. There may be no decoder at all if you use SA, as for example in BERT architecture.

3. SA may be applied many times independently within a single model (e.g. 18 times in Transformer, 12 times in BERT BASE) while AT is usually applied once in the model and connects some 2 components (e.g. encoder and decoder).

4. SA is good at modeling dependencies between different parts of the sequence. For example - understand the syntactic function between words in the sentence. AT on the other hand models only the dependencies between 2 different sequences (for example, the original text and the translation of the text). While still the SA may be very good in translation task (see Transformer)

5. AT can connect 2 different modalities (i.e. text and image). SA is usually applied within a single modality but you still can join activations from 2 modalities into a single sequence and apply SA on it.

6. Generally SA mechanism looks like a more general to me as it can do more than AT. You can simulate AT with SA by just replacing/concatenating the input sequence with the target sequence that you want your attention to look at.

Some more notes

• The term Multi Head Attention is often used with SA. But theoretically you can apply Multi Head approach to AT also.
• The following terms: content-base attention, additive attention, location base attention, general attention, dot-product attention, scaled dot-product attention - are used to describe different mechanisms of how inputs are multiplied/added together to get the attention score. All these mechanisms may be applied both to AT and SA.
• Key/Query/Value approach of attention calculation is usually applied to SA. But you can use it for AT also.

Answer 2

Let me try to keep it more intuitive and less mathematical

Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give out a final output which is 'supposed' to be something of a sequence embedding. This works well for short sequences but beyond a certain length, it starts 'forgetting' things.

To fix this, Bahdanau et al came up with a landmark paper in 2014. They used all the hidden states of the encoder (instead of just the last state) in the model at the decoder end. But the best part was - they made the model give particular 'attention' to certain hidden states when decoding each word. They let the model itself 'learn' which words to give Attention to and which ones to ignore during translation of each word at the decoder.

This approach worked brilliantly and for 4 years, various forms of attentions were proposed. RNN coupled with Attention seems to have solved a long pending problem in NLP.

Now the scene shifts to 2018 when a team at Google presented what was a gamechanger for NLP. The name of the paper was 'Attention is all you need' and they claimed that Attention was all that was needed to encode sequences. No RNNs and serial processing any more. Chuck out the LSTMs and GRU and just use attention for the encoding. Of course for this they made several changes to the way attention is applied. They used self-attention models which is largely inspired by a paper by Cheng et al https://arxiv.org/pdf/1601.06733.pdf. In self-attention, the concept of attention is used to encode sequences instead of RNNs. So both the encoder and decoder now dont have RNNs and instead use attention mechanisms. In itself simplest form - each word in the sequence attends to every other word in the same sequence and in this way relationship between words in the sequence are captured.

So to summarize the difference - traditional Attention was used in combination with RNNs to improve their performance. Self-attention is used INSTEAD OF RNNs and they do a much better job and are also much faster. So in that sense they are pretty different.

Answer 3

In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary statistic (weighted mean) of your data: \begin{align} c&=\mathbb{E}_p[V] \end{align} The big question is how you determine the elements of $p$, the probability vector. In classic attention, you have a domain $S$, which is a set of locations. Let's say we are in a language setting, in which case we have a sequence $X=\{x_1,\cdots,x_n\}$ of vector representations of words and we have domain $S=\{1,\cdots,n\}$. The weights $p$ are $p(t|X)$. For example, assume we have the sentence 'He ate all of the pies.' and we investigate $t=3$. The attention weight for $t=3$ essentially ask how important position $3$ is, given the whole sequence.

Self attention instead has the domain $S$ include both location/positional and observation value information, and instead of conditioning on the entire sequence, asks about the importance of one word (and its position), conditional on some other word and its location (vector representations of both), a query. This is $p(x_k|x_q)$. So instead of asking, 'how important is position $3$ given the sequence?', it asks 'how important is "all" at position $3$ given we have "pies" at position $6$'? Essentially you're replacing conditioning on the entire sequence with pairwise comparisons.