Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
3 answers
50 views

Why do we use similarity/cosine between Query and Key in attention?

Let's take an example sentence for translation: I am going to my home and play with toy house. For translating 'home', as per my understanding, Query will be 'house'...
Pratham's user avatar
0 votes
0 answers
12 views

How can self-attention be used to combine representations from long text?

The paper "How to Fine-Tune BERT for Text Classification?" discusses using self-attention to combine the representations of a long input text that has been broken into chunks (section 5.3.1)....
suse's user avatar
  • 3
2 votes
1 answer
24 views

Practical Experiments on Self-Attention Mechanisms: QQ^T vs. QK^T

I'm currently exploring the self-attention mechanism used in models like Transformers, and I have a question about the necessity of using a separate key matrix (K) instead of just using the query ...
Peyman's user avatar
  • 1,165
0 votes
0 answers
4 views

Intuition and technical explanation behind "3D-aware Feature Attention" in Sync Dreamer? (Multiview Generating Diffusion Model)

I am looking to understand the whole block on how the Sync Dreamer paper has constructed the "3D aware feature attention" which is outlined on page 5. To enforce consistency among multiple ...
ChaoS Adm's user avatar
  • 141
2 votes
0 answers
56 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
Mew's user avatar
  • 233
0 votes
0 answers
50 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
Reinis Mazeiks's user avatar
6 votes
2 answers
932 views

Cross-attention mask in Transformers

I can't fully understand how we should create the mask for the decoder's cross-attention mask in the original Transformer model from Attention Is All You Need. Here is my attempt at finding a solution:...
ИванКарамазов's user avatar
2 votes
0 answers
61 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
krise's user avatar
  • 121
0 votes
1 answer
63 views

Confused about Q and K in Attention Mechanism

The following equation computes the attention scores: A = softmax(QK / d) I think Q and K are interchangeable, but why is one called Query and the other called Key?
Matthew's user avatar
0 votes
1 answer
49 views

Understanding Multi-headed Attention from architecture details

I've a conceptual question BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture <...
Namburi Srinath's user avatar
0 votes
0 answers
11 views

What is the prior mu in Heterogeneous Graph Transformer?

I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
Servus's user avatar
  • 101
2 votes
3 answers
114 views

How can we use a transfomer model with new data if we still don't have the output?

Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms... But how can we use a transformer model to predict new ...
skan's user avatar
  • 185
0 votes
0 answers
36 views

Math Behind Additive Bahdanau Attention

I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism. The hidden state ...
user154214's user avatar
0 votes
1 answer
42 views

Tensorflow diagram for attention mechanism

I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture : which I don't understand. What do the ingoing ...
edamondo's user avatar
  • 101
2 votes
2 answers
825 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
0 votes
0 answers
93 views

Is it a good idea to use attention in VAEs for image generation?

There are research papers and codebases on GitHub that deal with VAEs for image generation on popular datasets like CelebA, etc. While surfing through Google Scholar I found self-attention and other ...
Sir Arthur7's user avatar
0 votes
0 answers
40 views

Are there other "interactive" non-linear neural network layers besides self-attention layer?

In the self-attention layer $$ \operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$ $Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
DeLorean88's user avatar
2 votes
3 answers
615 views

What does it mean order of input sequence does not matter for transformer self-attention head?

The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
DeLorean88's user avatar
0 votes
0 answers
39 views

Cross entropy loss starts out very low

I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For ...
Justin Goodrich's user avatar
0 votes
1 answer
260 views

query, key and value interpretation in transformers ( encoder - decoder framework )

I am implementing a custom algo inspired by NMT architecture BUT in the decoder, if Query = target language then the "value" should also be the same thing right ? Only the "key" ...
Vikram Murthy's user avatar
0 votes
1 answer
32 views

Why are they called Key,Value,Query-Vectors when they are actually matrices

Simple question really. People talk about K V Q Vectors when in a full attention all 3 are actually matrices (embed_size*sequence_length). So why the misnomer?
user2741831's user avatar
0 votes
1 answer
35 views

Query on Attention Architecture

As we most know that, Attention is focuses on specific parts of the input sequence those are most relevant in generating output sequence. Ex: The driver could not drive the car fast because it had a ...
tovijayak's user avatar
0 votes
0 answers
35 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
69 views

Transformers doubt

Basically here the $Q$,$K$ and $V$ are passed through a linear layer to obtain the actual $Q$,$K$ and $V$ for self attention mechanism and then we concatenate all of it. My doubt is, I thought the $Q$,...
NeverGiveUp's user avatar
1 vote
2 answers
2k views

computer vision transformers: ViT does not have a decoder?

from https://www.youtube.com/watch?v=TrdevFK_am4 that explains the paper titled, "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE" compare that to the architecture ...
mLstudent33's user avatar
0 votes
1 answer
168 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 1,165
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
1 vote
1 answer
79 views

Below text-classification model gives accuracy of 0.77 only on one dataset and 0.99 on spam-ham dataset? What should I do to increase with my dataset?

...
rutvi's user avatar
  • 19
1 vote
1 answer
853 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
dashnick's user avatar
  • 163
1 vote
2 answers
3k views

How many parameters does the vanilla Transformer have?

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
dopexxx's user avatar
  • 173
1 vote
1 answer
84 views

Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?

By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)  Still, I got a question from reading there, hopefully, you guys can help me out  Are the ...
EyeQ Tech's user avatar
  • 113
0 votes
1 answer
2k views

Vision Transformer ViT Parameter count

The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021) includes the following table: Can someone explain how they get the parameter counts or where my calculation is ...
Alex's user avatar
  • 111
1 vote
0 answers
89 views

Is normalization of word embeddings important?

I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
Ryan Keathley's user avatar
0 votes
1 answer
770 views

Is there bias in matrix multiplications for self attention

When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well. ...
butwhy's user avatar
  • 23
2 votes
1 answer
97 views

Transformer XL - understanding paper's illustration

If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where ...
AMT's user avatar
  • 121
1 vote
0 answers
237 views

Keras NLP TransformerDecoder MultiHeadAttention Value Error

Recently I have been working on a MIDI Music Generator using the TransformerEncoder & TransformerDecoder layers found in the Keras NLP library. There is not much info/help on these layers which is ...
Cameron A's user avatar
2 votes
2 answers
509 views

How do transformers differ from feature selection and regular machine learning?

This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
Socorro's user avatar
  • 111
0 votes
0 answers
25 views

Are Soft and Bahdanau attentions different?

I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like ...
Naveen Reddy Marthala's user avatar
0 votes
1 answer
943 views

Multi head self attention output size for batches with different sequence length

I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same ...
Murad's user avatar
  • 3
0 votes
1 answer
122 views

How to use strong labels in image classification?

I have a dataset where I have localized pixel-level annotations of a dataset of cancer vs non-cancer. Which deep learning methods can I use to optimize the model to focus on the localized regions of ...
Tirtha's user avatar
  • 33
5 votes
1 answer
1k views

Difference Between Attention and Fully Connected Layers in Deep Learning

There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
Adam's user avatar
  • 896
0 votes
0 answers
16 views

Does this kind of attention exist?

As someone who is new to deep learning, I am only familiar with self-attention. I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
user900476's user avatar
0 votes
1 answer
498 views

Transformers - Why Self Attention calculate dot product of q and k from of same word?

As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ ...
mon's user avatar
  • 721
0 votes
1 answer
542 views

Visualize attention area

I wonder how people draw a network's attention area on a single input. Such as: Any hint is much appreciated
zstr's user avatar
  • 13
1 vote
1 answer
2k views

How to add the Luong Attention Mechanism into CNN?

As I write my CNN model for an image binary classification below, I'm trying to add an attention layer to this model. I read from tf.keras.layers.Attention: https://...
xniwniw's user avatar
  • 11
1 vote
0 answers
188 views

Could Attention_mask in T5 be a float in [0,1]?

I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as ...
Dave's user avatar
  • 13
2 votes
0 answers
19 views

Custom Simulator for Deep Reinforcement Learning

I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
Esmaeel Mohammadi's user avatar
1 vote
1 answer
1k views

How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting

Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights. ...
Debashis Sahoo's user avatar
0 votes
0 answers
811 views

Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?

I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
Aushilfsgott's user avatar
0 votes
1 answer
199 views

For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq) But when using ...
Hank's user avatar
  • 3