Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5
votes
2answers
120 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
2answers
36 views

Splitting into multiple heads — multihead self attention

So, I have a doubt in Attention is all you need: The implementation of transformers on tensorflow's official documentation says: Each multi-head attention block gets three inputs; Q (query), K (key), ...
1
vote
0answers
18 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
3
votes
1answer
2k views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
9
votes
2answers
3k views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
0
votes
1answer
32 views

two different attention methods for seq2seq

I see two different ways of applying attention in seq2seq: (a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
43
votes
3answers
29k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
1
vote
1answer
24 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
1
vote
1answer
19 views

Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?

I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. ...
0
votes
0answers
13 views

Basic of the attention mechanism

Background Having gone through articles. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Attention in Neural Networks Visualizing A Neural Machine Translation Model (...
0
votes
1answer
18 views

Predicting point sequence in image

My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images. I am ...
0
votes
1answer
70 views

Attention model with seq2seq over sequence

On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps): ...
1
vote
1answer
50 views

How to add attention mechanism to my sequence-to-sequence architecture in Keras?

Based on this blog entry, I have written a sequence to sequence deep learning model in Keras: ...
0
votes
1answer
26 views

Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
1answer
34 views

Is a dense layer required for implementing Bahdanau attention?

I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed. This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
1
vote
1answer
29 views

Attention network without hidden state?

I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like: ...
0
votes
2answers
33 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
2
votes
3answers
944 views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
0
votes
1answer
27 views

Predict customer behaviour with Transformer(attention is all you need)

Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
1
vote
1answer
31 views

Attention to multiple areas of same sentence

Lets consider some sentences below: "Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too" "Can company1 buy company2? ...
3
votes
1answer
175 views

Attention for time-series in neural networks

Neural networks in many domains (audio, video, image text/NLP) can achieve great results. In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - ...
12
votes
3answers
631 views

How does attention mechanism learn?

I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding. I have this question because I’m ...
7
votes
2answers
4k views

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism ...
3
votes
2answers
121 views

Attention mechanism in Tensorflow 2

In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of ...
1
vote
2answers
170 views

Attention Mechanism: Why use context vector instead of attention weights?

In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the ...
1
vote
2answers
38 views

Role of decoder in Transformer?

I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
44 views

Why this TensorFlow Transformer model has Linear output instead of Softmax?

I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation. I am quite surprised that when the Transformer is created, their final output is a Dense ...
9
votes
3answers
1k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
2
votes
2answers
701 views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
0
votes
0answers
10 views

Using Transcoder Model for language to language conversion

I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
1
vote
1answer
41 views

Why does an attention layer in a transformer learn context?

I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers. What I'm confused on is why the ...
1
vote
1answer
450 views

How to add a Decoder & Attention Layer to Bidirectional Encoder with tensorflow 2.0

I am a beginner in machine learning and I'm trying to create a spelling correction model that spell checks for a small amount of vocab (approximately 1000 phrases). Currently, I am refering to the ...
0
votes
0answers
17 views

Question about Relative-Position-Representation code

In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py In _relative_attention_inner method, which I think is one of the ...
0
votes
1answer
28 views

Practical attention models

Attention is all you need is a nice paper that suggests using positional encodings as an alternative to RNNs in their Transformer architecture. GPT-2 and GPT-3 are examples of using this architecture ...
2
votes
1answer
1k views

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism. ...
1
vote
1answer
141 views

What would be the target input for Transformer Decoder during test phase?

The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer I am having confusion about what ...
1
vote
2answers
72 views

What are good toy problems for testing Transformer architectures?

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer ...
2
votes
1answer
55 views

Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow. I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline They implement the positional ...
2
votes
1answer
36 views

Transformer masking during training or inference?

I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
1
vote
0answers
36 views

What is the difference between additive and multiplicative attention? [closed]

This paper (https://arxiv.org/abs/1804.03999) implements additive addition. I think the attention module used in this paper (https://arxiv.org/abs/1805.08318) is an example of multiplicative attention,...
0
votes
2answers
76 views

What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
0
votes
1answer
69 views

NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

I am new to NLP and I just finished reading the paper "Attention is all you need". I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
3
votes
1answer
718 views

Transformer-based architectures for regression tasks

As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
1
vote
1answer
29 views

what is the difference between positional vector and attention vector used in transformer model?

what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
14
votes
3answers
6k views

Gumbel-Softmax trick vs Softmax with temperature

From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
0
votes
1answer
53 views

How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
2
votes
1answer
96 views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
0
votes
0answers
9 views

how many spectogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguisahble from humans) using the github https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
1
vote
2answers
41 views

In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?

In addition, why do we need a FFN in each layer when we already have attention? Here's a screenshot of the relevant section from Vaswani et al. (2017):
2
votes
2answers
501 views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...