Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
60
questions
1
vote
0answers
18 views
Is a dense layer required for implementing Bahdanau attention?
I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed.
This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
0
votes
0answers
13 views
Question about Relative-Position-Representation code
In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py
In _relative_attention_inner method, which I think is one of the ...
0
votes
1answer
24 views
Practical attention models
Attention is all you need is a nice paper that suggests using
positional encodings as an alternative to RNNs in their Transformer architecture.
GPT-2 and GPT-3 are examples of using this architecture ...
1
vote
1answer
46 views
What would be the target input for Transformer Decoder during test phase?
The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer
I am having confusion about what ...
1
vote
0answers
31 views
What is the difference between additive and multiplicative attention? [closed]
This paper (https://arxiv.org/abs/1804.03999) implements additive addition. I think the attention module used in this paper (https://arxiv.org/abs/1805.08318) is an example of multiplicative attention,...
2
votes
1answer
25 views
Transformer masking during training or inference?
I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
0
votes
1answer
22 views
Splitting into multiple heads — multihead self attention
So, I have a doubt in Attention is all you need:
The implementation of transformers on tensorflow's official documentation says:
Each multi-head attention block gets three inputs; Q (query), K (key), ...
0
votes
2answers
47 views
What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?
After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
2
votes
1answer
37 views
Explanation about i//2 in positional encoding in tensorflow tutorial about transformers
I was implementing the transformer architecture in tensorflow.
I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline
They implement the positional ...
0
votes
1answer
34 views
NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)
I am new to NLP and I just finished reading the paper "Attention is all you need".
I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
0
votes
0answers
54 views
Hierarchical transformer for document classification: error with model implementation, and advice on extracting attention weights
I am trying to implement a hierarchical transformer for document classification in Keras/tensorflow, in which:
(1) a word-level transformer produces a representation of each sentence, and attention ...
1
vote
1answer
24 views
what is the difference between positional vector and attention vector used in transformer model?
what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
0
votes
0answers
13 views
Predicting point sequence in image
My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images.
I am ...
0
votes
1answer
45 views
How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?
Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
3
votes
1answer
296 views
Transformer-based architectures for regression tasks
As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
2
votes
1answer
49 views
Transformer decoder output - how is it linear?
I'm not quite sure how's the decoder output is flattened into a single vector.
As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
0
votes
0answers
13 views
Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.
It ...
0
votes
1answer
194 views
How to add a Decoder & Attention Layer to Bidirectional Encoder with tensorflow 2.0
I am a beginner in machine learning and I'm trying to create a spelling correction model that spell checks for a small amount of vocab (approximately 1000 phrases). Currently, I am refering to the ...
1
vote
0answers
31 views
How to add attention mechanism to my sequence-to-sequence architecture in Keras?
Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:
...
0
votes
0answers
9 views
how many spectogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?
I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguisahble from humans) using the github https://github.com/Rayhane-mamah/Tacotron-2.
I'm very ...
1
vote
2answers
229 views
Does BERT use GLoVE?
From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated.
What I don't know ...
0
votes
0answers
17 views
Attention network without hidden state?
I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like:
...
1
vote
1answer
99 views
Can BERT be used for predicting words?
I have a question regarding the pre-training section (in particular, the Masked Language Model).
In the example Let's stick to improvisation in this skit, by masking the word improvisation, after ...
1
vote
2answers
40 views
What are good toy problems for testing Transformer architectures?
I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer ...
1
vote
2answers
35 views
In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?
In addition, why do we need a FFN in each layer when we already have attention?
Here's a screenshot of the relevant section from Vaswani et al. (2017):
0
votes
0answers
27 views
Attention model with seq2seq over sequence
On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps):
...
1
vote
1answer
29 views
Attention to multiple areas of same sentence
Lets consider some sentences below:
"Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too"
"Can company1 buy company2? ...
1
vote
1answer
507 views
How do Bahdanau - Luong Attentions use Query, Value, Key vectors?
In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
2
votes
1answer
452 views
Why does the transformer positional encoding use both sine and cosine?
In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.
I am wondering why it needs to use both sine and cosine though instead of just ...
0
votes
2answers
171 views
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
1
vote
1answer
163 views
What is the feedforward network in a transformer trained on?
After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns.
What I learned ...
2
votes
1answer
90 views
Attention mechanism in Tensorflow 2
In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of ...
10
votes
2answers
509 views
How does attention mechanism learn?
I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding.
I have this question because I’m ...
1
vote
2answers
2k views
Why is the decoder not a part of BERT architecture?
I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
1
vote
1answer
145 views
What is difference between attention mechanism and cognitive function?
How are attention mechanism used in different deep learning algorithms different from the cognitive function attention?
3
votes
1answer
2k views
SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors
I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
1
vote
0answers
105 views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
39 views
In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]
How many definitions of attention are commonly employed for Deep Learning tasks?
That's what I've encountered up to now:
Self-attention
Bahdanau
Luong
Multi-Head (used in Transformers)
Could you ...
1
vote
2answers
734 views
Any good Implementations of Bi-LSTM bahdanau attention in Keras?
From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
0answers
12 views
Training a model for Single Image Super Reoslution
I'm trying to implement the Attention-based approach for SISR paper. However, during something odd happens. The MAE for the first output of the model is very small. But as the training progresses, the ...
0
votes
0answers
13 views
How can we use Attention Layer in a Encoder Decoder Model Architecture for Next Phrase Prediction?
I am currently building a Encoder Decoder Model for Next Phrase Prediction, but only using this model is just giving satisfactory results. So I chose to attach a Attention Layer to the same ...
1
vote
2answers
1k views
What is the advantage of positional encoding over one hot encoding in a transformer model?
I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices.
In the paper (Section 3.5), they ...
0
votes
1answer
31 views
two different attention methods for seq2seq
I see two different ways of applying attention in seq2seq:
(a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
1
vote
0answers
51 views
How to Visualize Graph Attention
I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to ...
10
votes
3answers
5k views
Gumbel-Softmax trick vs Softmax with temperature
From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
9
votes
2answers
966 views
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
While reviewing the Transformer architecture, I realized something I didn't expect, which is that :
the positional encoding is summed to the word embeddings
rather than concatenated to it.
...
0
votes
1answer
129 views
Attention Mechanism: Why use context vector instead of attention weights?
In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the ...
2
votes
1answer
1k views
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.
...
34
votes
3answers
24k views
What is the positional encoding in the transformer model?
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:
I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
4
votes
1answer
3k views
What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?
As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism ...
