Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
104 votes
4 answers
111k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
Peyman's user avatar
  • 1,165
39 votes
8 answers
11k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
FremyCompany's user avatar
34 votes
3 answers
37k views

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism ...
Pratik.S's user avatar
  • 473
34 votes
4 answers
18k views

Gumbel-Softmax trick vs Softmax with temperature

From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
4-bit's user avatar
  • 441
29 votes
7 answers
25k views

Why is the decoder not a part of BERT architecture?

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
Hrishikesh Athalye's user avatar
21 votes
1 answer
12k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
CoderOnly's user avatar
  • 711
16 votes
3 answers
4k views

How does attention mechanism learn?

I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding. I have this question because I’m ...
user2790103's user avatar
14 votes
2 answers
12k views

Class token in ViT and BERT

I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT. To the best of my understanding this token is used to gather knowledge of the entire class, ...
Shir's user avatar
  • 241
13 votes
2 answers
9k views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
Sean Lee's user avatar
  • 251
12 votes
4 answers
7k views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
Milad Shahidi's user avatar
11 votes
2 answers
7k views

Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed. I am wondering why it needs to use both sine and cosine though instead of just ...
Joff's user avatar
  • 243
11 votes
2 answers
2k views

How do attention mechanisms in RNNs learn weights for a variable length input

Attention mechanisms in RNNs are reasonably common to sequence to sequence models. I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...
davidparks21's user avatar
8 votes
4 answers
9k views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
arctic_hen7's user avatar
7 votes
2 answers
5k views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
birdmw's user avatar
  • 173
7 votes
2 answers
5k views

Transformer-based architectures for regression tasks

As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
Damjan Dakic's user avatar
6 votes
2 answers
926 views

Cross-attention mask in Transformers

I can't fully understand how we should create the mask for the decoder's cross-attention mask in the original Transformer model from Attention Is All You Need. Here is my attempt at finding a solution:...
ИванКарамазов's user avatar
6 votes
1 answer
2k views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
Ian's user avatar
  • 63
5 votes
3 answers
996 views

What are good toy problems for testing Transformer architectures?

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time-consuming affair. What are good toy problems to test if a transformer ...
Christoph Henkelmann's user avatar
5 votes
1 answer
2k views

Attention for time-series in neural networks

Neural networks in many domains (audio, video, image text/NLP) can achieve great results. In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - ...
Georg Heiler's user avatar
5 votes
1 answer
8k views

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism. ...
Kahina's user avatar
  • 634
5 votes
1 answer
3k views

Transformer masking during training or inference?

I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
David Rein's user avatar
5 votes
1 answer
2k views

Why do position embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
Robin's user avatar
  • 1,337
5 votes
1 answer
1k views

Difference Between Attention and Fully Connected Layers in Deep Learning

There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
Adam's user avatar
  • 896
4 votes
2 answers
2k views

How is attention different from linear MLPs?

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
mohammad ali Humayun's user avatar
4 votes
1 answer
4k views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
Uday's user avatar
  • 556
4 votes
2 answers
2k views

Keras Attention Guided CNN problem

I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf In ...
Tenescu Andrei's user avatar
3 votes
1 answer
3k views

Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
J. Herrera's user avatar
3 votes
1 answer
1k views

What reccent alternatives to LSTM are there for regression problems?

I have been working for a while on a regression problem - predicting the air pollution in a city based on meteorological features (humidity, temperature, wind velocity a.o.). I have trained an LSTM ...
NeStack's user avatar
  • 238
3 votes
2 answers
647 views

Attention mechanism in Tensorflow 2

In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of ...
PKlumpp's user avatar
  • 147
3 votes
3 answers
3k views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
user_12's user avatar
  • 347
3 votes
1 answer
3k views

What exactly is the linear layer in the transformer model?

Please see this image: There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here: Are these linear layers ...
fac120's user avatar
  • 33
3 votes
1 answer
1k views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
Eka's user avatar
  • 301
3 votes
0 answers
244 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
cuuupid's user avatar
  • 131
3 votes
0 answers
751 views

How to train tensorflow's transformer model on my own data?

https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model. I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
Abhishek Niranjan's user avatar
2 votes
3 answers
8k views

What is the advantage of positional encoding over one hot encoding in a transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices. In the paper (Section 3.5), they ...
Inderpartap Cheema's user avatar
2 votes
3 answers
114 views

How can we use a transfomer model with new data if we still don't have the output?

Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms... But how can we use a transformer model to predict new ...
skan's user avatar
  • 185
2 votes
2 answers
821 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
2 votes
2 answers
509 views

How do transformers differ from feature selection and regular machine learning?

This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
Socorro's user avatar
  • 111
2 votes
3 answers
613 views

What does it mean order of input sequence does not matter for transformer self-attention head?

The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
DeLorean88's user avatar
2 votes
2 answers
2k views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
Leevo's user avatar
  • 6,255
2 votes
1 answer
24 views

Practical Experiments on Self-Attention Mechanisms: QQ^T vs. QK^T

I'm currently exploring the self-attention mechanism used in models like Transformers, and I have a question about the necessity of using a separate key matrix (K) instead of just using the query ...
Peyman's user avatar
  • 1,165
2 votes
1 answer
594 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
erre4's user avatar
  • 95
2 votes
2 answers
2k views

Role of decoder in Transformer?

I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
kyc12's user avatar
  • 165
2 votes
2 answers
153 views

Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow. I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline They implement the positional ...
Darome's user avatar
  • 33
2 votes
1 answer
97 views

Transformer XL - understanding paper's illustration

If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where ...
AMT's user avatar
  • 121
2 votes
1 answer
1k views

In Transformer's multi-headed attention, how attending "different representation subspaces at different positions" is achieved?

Question partially inspired by this post about the need of multi-head attention mechanism. For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(...
Deil's user avatar
  • 193
2 votes
1 answer
45 views

Nutritional image classification task

I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar ...
Josemafuen's user avatar
2 votes
2 answers
2k views

Attention Mechanism: Why use context vector instead of attention weights?

In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the ...
Physbox's user avatar
  • 217
2 votes
0 answers
56 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
Mew's user avatar
  • 233
2 votes
0 answers
61 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
krise's user avatar
  • 121