There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on or pay more attention to certain features, and has demonstrated some empirical success in NLP and related sequential models.
When I look at some code examples such as this one, adding an Attention layer intuitively makes sense and seems to improve performance of the LSTM model. However it looks very much like a regular fully-connected layer. In that link (and with some slight change of notation), the Attention layer outputs $$ c(x) = \tanh(\mathbf{W}x + \mathbf{b} ) $$ $$ \beta(x) = \frac{e^{c(x_j)}}{\sum_{j} e^{c(x_j)}} $$ $$ f_{Attention}(x) = x\beta $$ where $W,b$ are weights/biases, $x$ is layer input, and $f(.)$ is the layer output. In contrast, a regular fully connected layer: $$ f_{Dense} = \sigma(\mathbf{W}x + \mathbf{b}) $$ for some activation function $\sigma(.)$.
My interpretation of the Attention implementation above is that it is pretty much the same thing as a standard fully connected layer, but with a $\tanh$ activation (why?), followed by a $\text{softmax}$ (okay, so that the "attention weights" $\beta$ sum to 1), followed by a linear dot product.
How does this architecture allow the model to have "attention"? I do not see how it is fundamentally different or more expressive from just adding a standard fully-connected layer. Am I misunderstanding something here?
Edit/My Interpretation
I looked at the code example closer and there is in fact a difference due to the shapes of the parameters.
For an input $X$ with shape (N,T,K) where $N$ is samples, $T$ is time, and $K$ is features, the shapes of the Attention parameters are: $W$ (K, 1), $b$ (T,1). This implies that both the "context" $c$ and "attention weights" $\beta$ have shape (N,T,1). Thus in the final dot product, we are using the Attention layer to weight each timestep (e.g. pay more attention to prices yesterday and 5 days ago, but less attention to 30 days ago).
In contrast, a fully connected layer in this context would generate some (unnormalized) weights for each feature column uniformly across timesteps (e.g. pay more attention to all prices, but less to interest rates).
