How is attention different from linear MLPs?

Question

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?

Answer 1

@Jindřich mentioned

attention brings information about the context of the other vectors

I will include more description about how self-attention works in this way.

Let all word vectors in a sentence form a set $S=\{|\varphi_1\rangle, |\varphi_2\rangle,\cdots,|\varphi_m\rangle,\cdots,|\varphi_n\rangle\}$. If any two elements in set $S$ are orthogonal, we have

$|\varphi_m\rangle=\sum_{i=1}^n\delta_{im}|\varphi_i\rangle=\sum_{i=1}^n{|\varphi_i\rangle}{\langle\varphi_i|} {|\varphi_m\rangle}$,

where Kronecker delta $\delta_{i,m}=\langle\varphi_i|\varphi_m\rangle$ and $\sum_{i=1}^n|\varphi_i\rangle\langle\varphi_i|$ is an identity matrix.

If word vector $|\varphi_m\rangle$ wants to connect with context words in this sentence (e.g. pronouns connecting with other words, etc.), a possible approach is using a linear transformation to make elements in set $S$ not orthogonal anymore. Take single headed attention as an example, the matrixes related to the linear transformations are $\color{red}{W_q}$ , $\color{blue}{W_k}$ and $\color{purple}{W_v}$, the matrices of which are learned through backward propagation as weights. After linear transformation, the sets of word vectors are $\{\color{red}{|q_1\rangle}, \color{red}{|q_2\rangle},\cdots,\color{red}{|q_m\rangle},\cdots,\color{red}{|q_n\rangle}\}$, $\{\color{blue}{|k_1\rangle}, \color{blue}{|k_2\rangle},\cdots,\color{blue}{|k_m\rangle},\cdots,\color{blue}{|k_n\rangle}\}$ and $\{\color{purple}{|v_1\rangle}, \color{purple}{|v_2\rangle},\cdots,\color{purple}{|v_m\rangle},\cdots,\color{purple}{|v_n\rangle}\}$. Thus $|\varphi'_m\rangle\equiv\sum_{i=1}^n\color{purple}{|v_i\rangle}\color{blue}{\langle k_i|}\color{red}{|q_m\rangle}|$ as a new word vector contains the context information from other words in this sentence. If a word vector $|\varphi_m\rangle$ barely relates to other words in this sentence, then $|\varphi'_m\rangle\approx|\varphi_m\rangle$.

Note the new word vector here, $|\varphi'_m\rangle$, still has no clue about other word vectors' location information. To solve this issue, positional encoding is needed while forming set $S$ at the beginning.

Comparing $|\varphi'_m\rangle\equiv\sum_{i=1}^n\color{purple}{|v_i\rangle}\color{blue}{\langle k_i|}\color{red}{|q_m\rangle}$ and $|\varphi_m\rangle=\sum_{i=1}^n{|\varphi_i\rangle}{\langle\varphi_i|}{|\varphi_m\rangle}$, one may notice that $\color{purple}{|v_i\rangle}\color{blue}{\langle k_i|}$ is the new projection operator rather than $|\varphi_i\rangle\langle\varphi_i|$, and $\color{blue}{\langle k_i|}\color{red}{|q_m\rangle}$ is not necessarily equals to $\delta_{im}$ whereas $\langle\varphi_i|\varphi_m\rangle=\delta_{i,m}$.

Answer 2

The crucial difference here is that attention allows working with arbitrarily long sequences (or rather sets) of vectors.

A linear layer has a constant-sized input. Each output activation in a linear layer is a linear combination of the activations in the previous layer. However, the input is always exactly one vector, so linear layers cannot in principle consider any context. Processing a sequence with linear layers only is equivalent to processing each vector in the sequence independently. (A straightforward update would be doing a sliding window over the vector sequence, this is called 1D convolution.)

Attention can work with arbitrarily long input. It computes the similarity between a query vector with all key vectors and retrieves corresponding values. Unlike linear layers, attention brings information about the context of the other vectors. In the self-attention case, all vectors interact with each other.