Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
153
questions
0
votes
1
answer
20
views
Confused about Q and K in Attention Mechanism
The following equation computes the attention scores:
A = softmax(QK / d)
I think Q and K are interchangeable, but why is one called Query and the other called Key?
- 3
0
votes
1
answer
38
views
Understanding Multi-headed Attention from architecture details
I've a conceptual question
BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture
<...
0
votes
0
answers
9
views
What is the prior mu in Heterogeneous Graph Transformer?
I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
- 101
2
votes
3
answers
65
views
How can we use a transfomer model with new data if we still don't have the output?
Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms...
But how can we use a transformer model to predict new ...
- 185
0
votes
0
answers
31
views
Math Behind Additive Bahdanau Attention
I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism.
The hidden state ...
0
votes
1
answer
34
views
Tensorflow diagram for attention mechanism
I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :
which I don't understand. What do the ingoing ...
- 101
2
votes
2
answers
293
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
0
votes
0
answers
32
views
Is it a good idea to use attention in VAEs for image generation?
There are research papers and codebases on GitHub that deal with VAEs for image generation on popular datasets like CelebA, etc. While surfing through Google Scholar I found self-attention and other ...
- 101
0
votes
0
answers
34
views
Are there other "interactive" non-linear neural network layers besides self-attention layer?
In the self-attention layer
$$
\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V
$$
$Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
- 215
2
votes
3
answers
247
views
What does it mean order of input sequence does not matter for transformer self-attention head?
The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
- 215
0
votes
0
answers
25
views
Cross entropy loss starts out very low
I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For ...
0
votes
1
answer
112
views
query, key and value interpretation in transformers ( encoder - decoder framework )
I am implementing a custom algo inspired by NMT architecture BUT in the decoder, if Query = target language then the "value" should also be the same thing right ? Only the "key" ...
- 328
0
votes
1
answer
25
views
Why are they called Key,Value,Query-Vectors when they are actually matrices
Simple question really. People talk about K V Q Vectors when in a full attention all 3 are actually matrices (embed_size*sequence_length). So why the misnomer?
- 111
0
votes
1
answer
31
views
Query on Attention Architecture
As we most know that, Attention is focuses on specific parts of the input sequence those are most relevant in generating output sequence.
Ex: The driver could not drive the car fast because it had a ...
- 57
0
votes
0
answers
27
views
Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
0
votes
0
answers
47
views
val_accuracy and val_loss not changing while training transformer
recently i have been trying to learn transformer and using it in caption-generator model.
While training for 4 hours val_loss and val_accuracy did not change. loss and accuracy for train_data was ...
- 135
0
votes
0
answers
205
views
Keras masking with MultiHeadAttention
I am following keras example to classify time series using transformers. Timeseries classification with a Transformer model
The creation of the model is presented in the following code snippet:
...
0
votes
1
answer
59
views
Transformers doubt
Basically here the $Q$,$K$ and $V$ are passed through a linear layer to obtain the actual $Q$,$K$ and $V$ for self attention mechanism and then we concatenate all of it.
My doubt is, I thought the $Q$,...
0
votes
2
answers
915
views
computer vision transformers: ViT does not have a decoder?
from https://www.youtube.com/watch?v=TrdevFK_am4 that explains the paper titled, "AN IMAGE IS WORTH 16X16 WORDS:
TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE"
compare that to the architecture ...
- 574
0
votes
1
answer
89
views
In the attention mechanism, why don't we normalize after multiplying values?
As this question says:
In scaled dot product attention, we scale our outputs by dividing the
dot product by the square root of the dimensionality of the matrix:
The reason why is stated that this ...
- 1,113
1
vote
2
answers
770
views
Fine-tuned MLM based RoBERTa not improving performance
We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM.
We took some sample data points (2M+) & fine-tuned ...
- 11
1
vote
1
answer
50
views
1
vote
1
answer
539
views
Why does a decoder generate all hidden states during inference?
Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
- 163
0
votes
0
answers
20
views
Selecting an element in a sequence with self-attention networks
I have a doubt on I should set up the following problem:
Data:
My data is a tensor with shape (N, J, F) where N is the batch size, J is the sequence length, and F is the number of features of each ...
0
votes
2
answers
1k
views
How many parameters does the vanilla Transformer have?
The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
- 163
0
votes
0
answers
19
views
Training and Validation accuracy decreases after using more data
I have a binary classification project, I use a neural network with the following architecture:
The shape of the input is 64×64×4. This input was fed to a Conv2D layer with 32(5×5) filters followed by ...
0
votes
0
answers
37
views
In "Show, attend and tell", why do the attention weights get multiplied with the features to form the context vector?
The attention weights are formed through the last hidden state of the LSTM and the feature map from some kind of image encoder (in my case resnet so the features are in the form of 14x14x2048). They ...
- 101
1
vote
1
answer
75
views
Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?
By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)
Still, I got a question from reading there, hopefully, you guys can help me out
Are the ...
- 113
0
votes
0
answers
109
views
can I use tf.keras.layers.MultiHeadAttention for image classification task?
Could you please let me know whether it is possible to use tf.keras.layers.MultiHeadAttention() for the image classification task without using Vision Transformer techniqe?
0
votes
1
answer
1k
views
Vision Transformer ViT Parameter count
The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021)
includes the following table:
Can someone explain how they get the parameter counts or where my calculation is ...
- 111
0
votes
0
answers
49
views
Why do transformers operate on discrete sequences?
Why do we need to discretize our input $x$ vectors in transformers? For example, we often employ VQ-VAE's to discretize images to interface with ViTs.
Surely, because attention calculation simply ...
- 207
1
vote
0
answers
41
views
Is normalization of word embeddings important?
I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
0
votes
1
answer
455
views
Is there bias in matrix multiplications for self attention
When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well.
...
- 13
2
votes
1
answer
82
views
Transformer XL - understanding paper's illustration
If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where ...
- 121
1
vote
0
answers
215
views
Keras NLP TransformerDecoder MultiHeadAttention Value Error
Recently I have been working on a MIDI Music Generator using the TransformerEncoder & TransformerDecoder layers found in the Keras NLP library. There is not much info/help on these layers which is ...
- 11
1
vote
2
answers
355
views
How do transformers differ from feature selection and regular machine learning?
This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
- 101
0
votes
0
answers
24
views
Are Soft and Bahdanau attentions different?
I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like ...
0
votes
1
answer
659
views
Multi head self attention output size for batches with different sequence length
I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same ...
- 3
0
votes
1
answer
111
views
How to use strong labels in image classification?
I have a dataset where I have localized pixel-level annotations of a dataset of cancer vs non-cancer.
Which deep learning methods can I use to optimize the model to focus on the localized regions of ...
- 33
4
votes
1
answer
1k
views
Difference Between Attention and Fully Connected Layers in Deep Learning
There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
- 836
0
votes
0
answers
15
views
Does this kind of attention exist?
As someone who is new to deep learning, I am only familiar with self-attention.
I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
- 165
0
votes
1
answer
285
views
Transformers - Why Self Attention calculate dot product of q and k from of same word?
As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ ...
- 585
0
votes
1
answer
412
views
Visualize attention area
I wonder how people draw a network's attention area on a single input.
Such as:
Any hint is much appreciated
- 13
1
vote
1
answer
2k
views
How to add the Luong Attention Mechanism into CNN?
As I write my CNN model for an image binary classification below, I'm trying to add an attention layer to this model. I read from tf.keras.layers.Attention: https://...
- 11
1
vote
0
answers
177
views
Could Attention_mask in T5 be a float in [0,1]?
I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as
...
- 13
2
votes
0
answers
16
views
Custom Simulator for Deep Reinforcement Learning
I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
1
vote
1
answer
1k
views
How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting
Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights.
...
- 113
0
votes
0
answers
619
views
Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?
I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
0
votes
1
answer
160
views
For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?
The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See:
Why does LSTM performs better when the source target is reversed? (Seq2seq)
But when using ...
- 3
1
vote
1
answer
844
views
Positional encoding without input embedding
Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer?
In NLP models, the embedding maps a word to real numbers. ...
- 113