Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
853 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
1 vote
2 answers
2k views

Self Attention vs LSTM with Attention for NMT

I am trying to compare the A: Transformer-based architecture for Neural Machine Translation (NMT) from the Attention is All You Need paper, with B: an architecture based on Bi-directional LSTM's in ...
0 votes
1 answer
343 views

Attention model with seq2seq over sequence

On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps): ...
2 votes
1 answer
231 views

Why does Keras only have 3 types of attention layers?

The Keras library list only has 3 types of attentions - keras attention layers, which are : MultiHeadAttention layer Attention layer AdditiveAttention layer However, in theory there are multiple ...
0 votes
2 answers
712 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
0 votes
3 answers
50 views

Why do we use similarity/cosine between Query and Key in attention?

Let's take an example sentence for translation: I am going to my home and play with toy house. For translating 'home', as per my understanding, Query will be 'house'...
2 votes
1 answer
24 views

Practical Experiments on Self-Attention Mechanisms: QQ^T vs. QK^T

I'm currently exploring the self-attention mechanism used in models like Transformers, and I have a question about the necessity of using a separate key matrix (K) instead of just using the query ...
1 vote
2 answers
556 views

How do attention mechanism in CNN for images?

I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the ...
0 votes
0 answers
12 views

How can self-attention be used to combine representations from long text?

The paper "How to Fine-Tune BERT for Text Classification?" discusses using self-attention to combine the representations of a long input text that has been broken into chunks (section 5.3.1)....
1 vote
1 answer
64 views

Predicting point sequence in image

My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images. I am ...
0 votes
1 answer
770 views

Is there bias in matrix multiplications for self attention

When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well. ...
1 vote
1 answer
61 views

Attention network without hidden state?

I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like: ...
1 vote
2 answers
303 views

Is a dense layer required for implementing Bahdanau attention?

I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed. This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
0 votes
0 answers
4 views

Intuition and technical explanation behind "3D-aware Feature Attention" in Sync Dreamer? (Multiview Generating Diffusion Model)

I am looking to understand the whole block on how the Sync Dreamer paper has constructed the "3D aware feature attention" which is outlined on page 5. To enforce consistency among multiple ...
12 votes
4 answers
7k views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
2 votes
1 answer
680 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
1 vote
1 answer
1k views

How to add a Decoder & Attention Layer to Bidirectional Encoder with tensorflow 2.0

I am a beginner in machine learning and I'm trying to create a spelling correction model that spell checks for a small amount of vocab (approximately 1000 phrases). Currently, I am refering to the ...
0 votes
1 answer
168 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
1 vote
2 answers
2k views

computer vision transformers: ViT does not have a decoder?

from https://www.youtube.com/watch?v=TrdevFK_am4 that explains the paper titled, "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE" compare that to the architecture ...
1 vote
1 answer
74 views

two different attention methods for seq2seq

I see two different ways of applying attention in seq2seq: (a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
0 votes
2 answers
332 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
2 votes
3 answers
8k views

What is the advantage of positional encoding over one hot encoding in a transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices. In the paper (Section 3.5), they ...
39 votes
8 answers
11k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
1 vote
1 answer
113 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
8 votes
4 answers
9k views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
0 votes
2 answers
156 views

Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
5 votes
1 answer
1k views

Difference Between Attention and Fully Connected Layers in Deep Learning

There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
4 votes
2 answers
2k views

How is attention different from linear MLPs?

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
2 votes
0 answers
56 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
0 votes
0 answers
50 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
6 votes
2 answers
933 views

Cross-attention mask in Transformers

I can't fully understand how we should create the mask for the decoder's cross-attention mask in the original Transformer model from Attention Is All You Need. Here is my attempt at finding a solution:...
2 votes
2 answers
153 views

Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow. I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline They implement the positional ...
2 votes
0 answers
61 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
2 votes
2 answers
509 views

How do transformers differ from feature selection and regular machine learning?

This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
0 votes
1 answer
63 views

Confused about Q and K in Attention Mechanism

The following equation computes the attention scores: A = softmax(QK / d) I think Q and K are interchangeable, but why is one called Query and the other called Key?
2 votes
3 answers
615 views

What does it mean order of input sequence does not matter for transformer self-attention head?

The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
0 votes
1 answer
49 views

Understanding Multi-headed Attention from architecture details

I've a conceptual question BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture <...
0 votes
1 answer
260 views

query, key and value interpretation in transformers ( encoder - decoder framework )

I am implementing a custom algo inspired by NMT architecture BUT in the decoder, if Query = target language then the "value" should also be the same thing right ? Only the "key" ...
1 vote
2 answers
3k views

How many parameters does the vanilla Transformer have?

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
0 votes
0 answers
11 views

What is the prior mu in Heterogeneous Graph Transformer?

I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
2 votes
3 answers
114 views

How can we use a transfomer model with new data if we still don't have the output?

Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms... But how can we use a transformer model to predict new ...
11 votes
2 answers
7k views

Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed. I am wondering why it needs to use both sine and cosine though instead of just ...
1 vote
1 answer
228 views

Multioutput prediction using LSTM encoder decoder with Attention

(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.) I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
0 votes
0 answers
36 views

Math Behind Additive Bahdanau Attention

I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism. The hidden state ...
0 votes
1 answer
42 views

Tensorflow diagram for attention mechanism

I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture : which I don't understand. What do the ingoing ...
2 votes
2 answers
825 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
0 votes
0 answers
93 views

Is it a good idea to use attention in VAEs for image generation?

There are research papers and codebases on GitHub that deal with VAEs for image generation on popular datasets like CelebA, etc. While surfing through Google Scholar I found self-attention and other ...
0 votes
0 answers
40 views

Are there other "interactive" non-linear neural network layers besides self-attention layer?

In the self-attention layer $$ \operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$ $Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
0 votes
0 answers
39 views

Cross entropy loss starts out very low

I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For ...