Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?


The crucial difference here is that attention allows working with arbitrarily long sequences (or rather sets) of vectors.

A linear layer has a constant-sized input. Each output activation in a linear layer is a linear combination of the activations in the previous layer. However, the input is always exactly one vector, so linear layers cannot in principle consider any context. Processing a sequence with linear layers only is equivalent to processing each vector in the sequence independently. (A straightforward update would be doing a sliding window over the vector sequence, this is called 1D convolution.)

Linear layers

Attention can work with arbitrarily long input. It computes the similarity between a query vector with all key vectors and retrieves corresponding values. Unlike linear layers, attention brings information about the context of the other vectors. In the self-attention case, all vectors interact with each other.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.