Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
91
questions
0
votes
0answers
10 views
Can the attention mask hold values between 0 and 1?
I am new to attention-based models and wanted to understand more about the attention mask in NLP models.
attention_mask: an optional torch.LongTensor of shape [...
0
votes
0answers
6 views
Visualization of SeqSelfAttention layer from Keras
Can someone help me to find, explain and visualize the SeqSelfAttention layer from Keras. I found a lot of flowchart that use the figures from the "attention is all you need paper" where ...
1
vote
0answers
14 views
A simple attention based text prediction model from scratch using pytorch
I first asked this question in codereview SE but a user recommended to post this here instead.
I have created a simple self attention based text prediction model using pytorch. The attention formula ...
2
votes
0answers
21 views
How does attention for feature fusion works
I am struggling to understand how would a self-attention layer be used for features of different modalities fusion.
What I understand until now is that :
Every unique modality is fed into a self-...
0
votes
0answers
11 views
Output shapes of Keras AdditiveAttention Layer
Trying to use the AdditiveAttention layer in Keras. On manual implementation of the layer from tensorflow tutorial https://www....
0
votes
0answers
17 views
Feature fusion multiple modalities example
I am trying to replicate this paper. In their architecture, there is an attention-fusion subnet explained as follows:
In the multimodal activity recognition problem, not all modalities are
equally ...
0
votes
0answers
14 views
Feature fusion with attention layer
I have three different modalities, each of which has a shape of (n-users, m-timestep, p-features). I want to do feature level fusion, by adding an attention layer that would compute the scores of ...
1
vote
1answer
34 views
Why are convolutions still used in some Transformer networks for speech enhancement?
So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...
2
votes
1answer
55 views
Does multi-head attention remove the need for self-attention?
The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
0
votes
0answers
23 views
How does multi-head attention on “multiple attention axes” works?
I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a ...
0
votes
2answers
29 views
Intuition of “Head” in Attention models (Transformer)?
I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!
1
vote
0answers
54 views
How to use in built Keras ADDITIVE ATTENTION Layer for image captioning?
I have Designed an Encoder-Decoder Model for Image Captioning. Now, I want to improve my Model. So, I thought of putting an Attention Layer in my Encoder-Decoder model. But, I am struggling with how ...
1
vote
0answers
19 views
SAGAN - what is the correct architecture?
Hi, in the original paper the following scheme of the self-attention appears:
https://arxiv.org/pdf/1805.08318.pdf
In a later overview:
https://arxiv.org/pdf/1906.01529.pdf
this scheme appears:
...
0
votes
0answers
54 views
Why do relative positional embeddings instead of absolute positional encoding improve the Transformer?
I've been researching the Music Transformer, the paper for which introduced an efficient algorithm to compute Relative Positional Embeddings in a Transformer. I know that Relative Positional ...
0
votes
1answer
13 views
There could be a problem with the linear layer after the attention inside a transformer?
My question regards this image:
It seems that after the multi head attention there is a linear layer as they mention also from here:
the linearity is given by the weights W^{o}. my quesion is: for ...
0
votes
1answer
300 views
Class token in ViT and BERT
I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT.
To the best of my understanding this token is used to gather knowledge of the entire class, ...
1
vote
1answer
63 views
Decoder Transformer feedforward
I have a question about the decoder transformer feed forward during training.
Let's pick an example: input data "i love the sun" traduction i want to ...
0
votes
1answer
40 views
Attention in NER
I am building a named entity recognition model and it is having BERT+BiLSTM+CRF in it. Now I am planning to introduce an attention layer. My question is - what type of attention I should use here and ...
0
votes
0answers
12 views
Language Model with Attention not learning
Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7%
The way input data is pipelined is by ...
1
vote
1answer
80 views
How do attention mechanism in CNN for images?
I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the ...
0
votes
1answer
20 views
Transformer architecture question
I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/.
The first attention ...
2
votes
1answer
37 views
Nutritional image classification task
I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar ...
0
votes
1answer
58 views
How do the linear layers in the attention mechanism work?
I think I now the answer to my question but I dont really get confirmation.
When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
1
vote
0answers
25 views
how is the linear relation between positional encoding helping attention?
I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
7
votes
2answers
474 views
Transformer model: Why are word embeddings scaled before adding positional encodings?
While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
1answer
42 views
Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?
I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. ...
0
votes
0answers
23 views
Basic of the attention mechanism
Background
Having gone through articles.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Attention in Neural Networks
Visualizing A Neural Machine Translation Model (...
0
votes
1answer
39 views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
3
votes
1answer
386 views
Attention for time-series in neural networks
Neural networks in many domains (audio, video, image text/NLP) can achieve great results.
In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - ...
1
vote
2answers
107 views
Role of decoder in Transformer?
I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
94 views
Why this TensorFlow Transformer model has Linear output instead of Softmax?
I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation.
I am quite surprised that when the Transformer is created, their final output is a Dense ...
0
votes
0answers
10 views
Using Transcoder Model for language to language conversion
I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
1
vote
1answer
28 views
Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?
So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
1
vote
1answer
67 views
Why does an attention layer in a transformer learn context?
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the ...
0
votes
2answers
49 views
Understanding Transformer's Self attention calculations
I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
1
vote
1answer
45 views
Is a dense layer required for implementing Bahdanau attention?
I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed.
This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
0
votes
0answers
23 views
Question about Relative-Position-Representation code
In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py
In _relative_attention_inner method, which I think is one of the ...
0
votes
1answer
29 views
Practical attention models
Attention is all you need is a nice paper that suggests using
positional encodings as an alternative to RNNs in their Transformer architecture.
GPT-2 and GPT-3 are examples of using this architecture ...
1
vote
1answer
408 views
What would be the target input for Transformer Decoder during test phase?
The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer
I am having confusion about what ...
1
vote
0answers
67 views
What is the difference between additive and multiplicative attention? [closed]
This paper (https://arxiv.org/abs/1804.03999) implements additive addition. I think the attention module used in this paper (https://arxiv.org/abs/1805.08318) is an example of multiplicative attention,...
2
votes
1answer
92 views
Transformer masking during training or inference?
I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
1
vote
2answers
181 views
Splitting into multiple heads — multihead self attention
So, I have a doubt in Attention is all you need:
The implementation of transformers on tensorflow's official documentation says:
Each multi-head attention block gets three inputs; Q (query), K (key), ...
0
votes
2answers
146 views
What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?
After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
2
votes
1answer
82 views
Explanation about i//2 in positional encoding in tensorflow tutorial about transformers
I was implementing the transformer architecture in tensorflow.
I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline
They implement the positional ...
0
votes
1answer
118 views
NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)
I am new to NLP and I just finished reading the paper "Attention is all you need".
I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
1
vote
1answer
36 views
what is the difference between positional vector and attention vector used in transformer model?
what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
1
vote
1answer
21 views
Predicting point sequence in image
My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images.
I am ...
0
votes
1answer
83 views
How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?
Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
5
votes
1answer
2k views
Transformer-based architectures for regression tasks
As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
5
votes
1answer
190 views
Transformer decoder output - how is it linear?
I'm not quite sure how's the decoder output is flattened into a single vector.
As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
