This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the training goes on.
How does this differ from regular feature selection and neural network training on a subset of the input?
Your question does not necessarily apply to transformers but to machine learning in general.
A question I can answer is: What is the difference between feature selection and machine learning?
The distinction is that humans might not know what features are important, and algorithms may also not be able to understand what information is relevant, so we let the model attend to what it finds important. (What it finds important is implicitly learned through the training objective).
Side Note: What a model finds important can be affected by its architecture.
Convolutional Neural Networks have a locality bias because they use kernels that run over the input, calculating a value based on other nearby values. While transformers have no bias, and attend to everything equaly (again due to the architecture, i.e. how values are calculated). If your task requires looking at the importance of nearby values then a CNN might train faster and need fewer data since it already has the bias, while the transformer will have to learn it.
Background
Although it is a prerequisite to Transformer, attention mechanism was introduced four years earlier 1 as part of an improved LSTM architecture.
Attention Pooling Layers
Attention improves performance of recurrent architectures by allowing the model to reason, not only from hidden state of the final position (Figure - top), but also from any position in the sequence input independently of how far back it may be (Figure - bottom). In other words, we could call this an extension of the local LSTM attention to an attention mechanism that allows for much much longer sequences to be modelled. The fixed
Attention in Transformers
In transformers the self-attention mechanism is leveraged (Wang et al., also proposed prior to the transformer architecture) so that the recurrence in layers is completely abandoned with a positional encoding taking its role for ensuring/encoding the position/order of each input token.
But imo, the key feature of the transformer is the parallel computation exactly because you don't anymore have to loop through each event of the sequential input to impose the chronological/stepwise ordering of tokens.
Source: Lmk if you identify the source of this figure, as I don't recall it
[1] Neural Machine Translation by Jointly Learning to Align and Translate. https://arxiv.org/pdf/1409.0473.pdf
