There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on or pay more attention to certain features, and has demonstrated some empirical success in NLP and related sequential models.

When I look at some code examples such as this one, adding an Attention layer intuitively makes sense and seems to improve performance of the LSTM model. However it looks very much like a regular fully-connected layer. In that link (and with some slight change of notation), the Attention layer outputs $$ c(x) = \tanh(\mathbf{W}x + \mathbf{b} ) $$ $$ \beta(x) = \frac{e^{c(x_j)}}{\sum_{j} e^{c(x_j)}} $$ $$ f_{Attention}(x) = x\beta $$ where $W,b$ are weights/biases, $x$ is layer input, and $f(.)$ is the layer output. In contrast, a regular fully connected layer: $$ f_{Dense} = \sigma(\mathbf{W}x + \mathbf{b}) $$ for some activation function $\sigma(.)$.

My interpretation of the Attention implementation above is that it is pretty much the same thing as a standard fully connected layer, but with a $\tanh$ activation (why?), followed by a $\text{softmax}$ (okay, so that the "attention weights" $\beta$ sum to 1), followed by a linear dot product.

How does this architecture allow the model to have "attention"? I do not see how it is fundamentally different or more expressive from just adding a standard fully-connected layer. Am I misunderstanding something here?

Edit/My Interpretation

I looked at the code example closer and there is in fact a difference due to the shapes of the parameters.

For an input $X$ with shape (N,T,K) where $N$ is samples, $T$ is time, and $K$ is features, the shapes of the Attention parameters are: $W$ (K, 1), $b$ (T,1). This implies that both the "context" $c$ and "attention weights" $\beta$ have shape (N,T,1). Thus in the final dot product, we are using the Attention layer to weight each timestep (e.g. pay more attention to prices yesterday and 5 days ago, but less attention to 30 days ago).

In contrast, a fully connected layer in this context would generate some (unnormalized) weights for each feature column uniformly across timesteps (e.g. pay more attention to all prices, but less to interest rates).


1 Answer 1


The attention mechanism is based on transformers that also include fully connected layers indeed.

The difference comes mainly from the RNNs that are the base of the attention mechanism. However, this attention is quite limited.

It could be difficult to understand how it works precisely, even if you get the idea step by step.

That's why, a good way to understand how they work is to use playgrounds like those ones:



  • $\begingroup$ Does it answer your question? If not, please let me know. $\endgroup$ Aug 23, 2022 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.