Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

Confused about Q and K in Attention Mechanism

The following equation computes the attention scores: A = softmax(QK / d) I think Q and K are interchangeable, but why is one called Query and the other called Key?
Matthew's user avatar
0 votes
1 answer
38 views

Understanding Multi-headed Attention from architecture details

I've a conceptual question BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture <...
Namburi Srinath's user avatar
0 votes
0 answers
9 views

What is the prior mu in Heterogeneous Graph Transformer?

I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
Servus's user avatar
  • 101
2 votes
3 answers
65 views

How can we use a transfomer model with new data if we still don't have the output?

Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms... But how can we use a transformer model to predict new ...
skan's user avatar
  • 185
0 votes
0 answers
31 views

Math Behind Additive Bahdanau Attention

I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism. The hidden state ...
user154214's user avatar
0 votes
1 answer
34 views

Tensorflow diagram for attention mechanism

I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture : which I don't understand. What do the ingoing ...
edamondo's user avatar
  • 101
2 votes
2 answers
293 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
0 votes
0 answers
32 views

Is it a good idea to use attention in VAEs for image generation?

There are research papers and codebases on GitHub that deal with VAEs for image generation on popular datasets like CelebA, etc. While surfing through Google Scholar I found self-attention and other ...
Sir Arthur7's user avatar
0 votes
0 answers
34 views

Are there other "interactive" non-linear neural network layers besides self-attention layer?

In the self-attention layer $$ \operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$ $Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
DeLorean88's user avatar
2 votes
3 answers
247 views

What does it mean order of input sequence does not matter for transformer self-attention head?

The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
DeLorean88's user avatar
0 votes
0 answers
25 views

Cross entropy loss starts out very low

I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For ...
Justin Goodrich's user avatar
0 votes
1 answer
112 views

query, key and value interpretation in transformers ( encoder - decoder framework )

I am implementing a custom algo inspired by NMT architecture BUT in the decoder, if Query = target language then the "value" should also be the same thing right ? Only the "key" ...
Vikram Murthy's user avatar
0 votes
1 answer
25 views

Why are they called Key,Value,Query-Vectors when they are actually matrices

Simple question really. People talk about K V Q Vectors when in a full attention all 3 are actually matrices (embed_size*sequence_length). So why the misnomer?
user2741831's user avatar
0 votes
1 answer
31 views

Query on Attention Architecture

As we most know that, Attention is focuses on specific parts of the input sequence those are most relevant in generating output sequence. Ex: The driver could not drive the car fast because it had a ...
tovijayak's user avatar
0 votes
0 answers
27 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
47 views

val_accuracy and val_loss not changing while training transformer

recently i have been trying to learn transformer and using it in caption-generator model. While training for 4 hours val_loss and val_accuracy did not change. loss and accuracy for train_data was ...
tikendraw's user avatar
  • 135
0 votes
0 answers
205 views

Keras masking with MultiHeadAttention

I am following keras example to classify time series using transformers. Timeseries classification with a Transformer model The creation of the model is presented in the following code snippet: ...
nicos prive's user avatar
0 votes
1 answer
59 views

Transformers doubt

Basically here the $Q$,$K$ and $V$ are passed through a linear layer to obtain the actual $Q$,$K$ and $V$ for self attention mechanism and then we concatenate all of it. My doubt is, I thought the $Q$,...
NeverGiveUp's user avatar
0 votes
2 answers
915 views

computer vision transformers: ViT does not have a decoder?

from https://www.youtube.com/watch?v=TrdevFK_am4 that explains the paper titled, "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE" compare that to the architecture ...
mLstudent33's user avatar
0 votes
1 answer
89 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 1,113
1 vote
2 answers
770 views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
1 vote
1 answer
50 views

Below text-classification model gives accuracy of 0.77 only on one dataset and 0.99 on spam-ham dataset? What should I do to increase with my dataset?

...
rutvi's user avatar
  • 19
1 vote
1 answer
539 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
dashnick's user avatar
  • 163
0 votes
0 answers
20 views

Selecting an element in a sequence with self-attention networks

I have a doubt on I should set up the following problem: Data: My data is a tensor with shape (N, J, F) where N is the batch size, J is the sequence length, and F is the number of features of each ...
Piero Viscone's user avatar
0 votes
2 answers
1k views

How many parameters does the vanilla Transformer have?

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
dopexxx's user avatar
  • 163
0 votes
0 answers
19 views

Training and Validation accuracy decreases after using more data

I have a binary classification project, I use a neural network with the following architecture: The shape of the input is 64×64×4. This input was fed to a Conv2D layer with 32(5×5) filters followed by ...
Pranshul Lakhanpal's user avatar
0 votes
0 answers
37 views

In "Show, attend and tell", why do the attention weights get multiplied with the features to form the context vector?

The attention weights are formed through the last hidden state of the LSTM and the feature map from some kind of image encoder (in my case resnet so the features are in the form of 14x14x2048). They ...
Fenrir's user avatar
  • 101
1 vote
1 answer
75 views

Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?

By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)  Still, I got a question from reading there, hopefully, you guys can help me out  Are the ...
EyeQ Tech's user avatar
  • 113
0 votes
0 answers
109 views

can I use tf.keras.layers.MultiHeadAttention for image classification task?

Could you please let me know whether it is possible to use tf.keras.layers.MultiHeadAttention() for the image classification task without using Vision Transformer techniqe?
code_lover's user avatar
0 votes
1 answer
1k views

Vision Transformer ViT Parameter count

The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021) includes the following table: Can someone explain how they get the parameter counts or where my calculation is ...
Alex's user avatar
  • 111
0 votes
0 answers
49 views

Why do transformers operate on discrete sequences?

Why do we need to discretize our input $x$ vectors in transformers? For example, we often employ VQ-VAE's to discretize images to interface with ViTs. Surely, because attention calculation simply ...
neel g's user avatar
  • 207
1 vote
0 answers
41 views

Is normalization of word embeddings important?

I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
Ryan Keathley's user avatar
0 votes
1 answer
455 views

Is there bias in matrix multiplications for self attention

When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well. ...
butwhy's user avatar
  • 13
2 votes
1 answer
82 views

Transformer XL - understanding paper's illustration

If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where ...
AMT's user avatar
  • 121
1 vote
0 answers
215 views

Keras NLP TransformerDecoder MultiHeadAttention Value Error

Recently I have been working on a MIDI Music Generator using the TransformerEncoder & TransformerDecoder layers found in the Keras NLP library. There is not much info/help on these layers which is ...
Cameron A's user avatar
1 vote
2 answers
355 views

How do transformers differ from feature selection and regular machine learning?

This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
Socorro's user avatar
  • 101
0 votes
0 answers
24 views

Are Soft and Bahdanau attentions different?

I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like ...
Naveen Reddy Marthala's user avatar
0 votes
1 answer
659 views

Multi head self attention output size for batches with different sequence length

I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same ...
Murad's user avatar
  • 3
0 votes
1 answer
111 views

How to use strong labels in image classification?

I have a dataset where I have localized pixel-level annotations of a dataset of cancer vs non-cancer. Which deep learning methods can I use to optimize the model to focus on the localized regions of ...
Tirtha's user avatar
  • 33
4 votes
1 answer
1k views

Difference Between Attention and Fully Connected Layers in Deep Learning

There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
Adam's user avatar
  • 836
0 votes
0 answers
15 views

Does this kind of attention exist?

As someone who is new to deep learning, I am only familiar with self-attention. I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
user900476's user avatar
0 votes
1 answer
285 views

Transformers - Why Self Attention calculate dot product of q and k from of same word?

As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ ...
mon's user avatar
  • 585
0 votes
1 answer
412 views

Visualize attention area

I wonder how people draw a network's attention area on a single input. Such as: Any hint is much appreciated
zstr's user avatar
  • 13
1 vote
1 answer
2k views

How to add the Luong Attention Mechanism into CNN?

As I write my CNN model for an image binary classification below, I'm trying to add an attention layer to this model. I read from tf.keras.layers.Attention: https://...
xniwniw's user avatar
  • 11
1 vote
0 answers
177 views

Could Attention_mask in T5 be a float in [0,1]?

I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as ...
Dave's user avatar
  • 13
2 votes
0 answers
16 views

Custom Simulator for Deep Reinforcement Learning

I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
Esmaeel Mohammadi's user avatar
1 vote
1 answer
1k views

How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting

Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights. ...
Debashis Sahoo's user avatar
0 votes
0 answers
619 views

Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?

I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
Aushilfsgott's user avatar
0 votes
1 answer
160 views

For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq) But when using ...
Hank's user avatar
  • 3
1 vote
1 answer
844 views

Positional encoding without input embedding

Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer? In NLP models, the embedding maps a word to real numbers. ...
Steven Morad's user avatar