Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [attention-mechanism]

The tag has no usage guidance.

53 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
1 answer
1k views

Difference Between Attention and Fully Connected Layers in Deep Learning

There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
Adam's user avatar
  • 896
3 votes
0 answers
244 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
cuuupid's user avatar
  • 131
3 votes
0 answers
751 views

How to train tensorflow's transformer model on my own data?

https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model. I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
Abhishek Niranjan's user avatar
2 votes
0 answers
56 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
Mew's user avatar
  • 233
2 votes
0 answers
61 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
krise's user avatar
  • 121
2 votes
0 answers
19 views

Custom Simulator for Deep Reinforcement Learning

I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
Esmaeel Mohammadi's user avatar
2 votes
0 answers
213 views

How to use the keras.layers.AdditiveAttention correctly?

My understanding on the topic is superficial at best, so do bear with me. I have a couple questions (specifically on how to use keras.layers.AdditiveAttention) which I hope is suitable to be asked ...
mathnoob's user avatar
  • 193
2 votes
1 answer
231 views

Why does Keras only have 3 types of attention layers?

The Keras library list only has 3 types of attentions - keras attention layers, which are : MultiHeadAttention layer Attention layer AdditiveAttention layer However, in theory there are multiple ...
Sandeep Bhutani's user avatar
2 votes
1 answer
680 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
neel g's user avatar
  • 227
2 votes
0 answers
525 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
boredaf's user avatar
  • 161
2 votes
0 answers
502 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
Shivam...'s user avatar
  • 123
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
1 vote
1 answer
853 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
dashnick's user avatar
  • 163
1 vote
0 answers
89 views

Is normalization of word embeddings important?

I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
Ryan Keathley's user avatar
1 vote
0 answers
237 views

Keras NLP TransformerDecoder MultiHeadAttention Value Error

Recently I have been working on a MIDI Music Generator using the TransformerEncoder & TransformerDecoder layers found in the Keras NLP library. There is not much info/help on these layers which is ...
Cameron A's user avatar
1 vote
0 answers
188 views

Could Attention_mask in T5 be a float in [0,1]?

I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as ...
Dave's user avatar
  • 13
1 vote
0 answers
83 views

Why does Bahdanau Attention Have to be Causal?

Using the Bahdanau attention layer on Tensorflow for time series prediction, although conceptually it is similar to NLP applications. This is how the minimal example code for a single layer looks like....
Della's user avatar
  • 335
1 vote
2 answers
2k views

Self Attention vs LSTM with Attention for NMT

I am trying to compare the A: Transformer-based architecture for Neural Machine Translation (NMT) from the Attention is All You Need paper, with B: an architecture based on Bi-directional LSTM's in ...
mashrivas's user avatar
1 vote
0 answers
34 views

Can a reformer model really handle long-range dependency?

I read this article about new attention model called Reformer. Here is the main strength of this model: The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
Kenenbek Arzymatov's user avatar
1 vote
0 answers
47 views

Which layer actually contains attention and how to map with input for classification scenario

I am writing a binary classification case and want to use Attention. The code is here - Code and model graph is shown below with attention area highlighted . I want to visualize which which word is ...
Sandeep Bhutani's user avatar
1 vote
0 answers
46 views

Why do we need dot product as part of the Transformer's training process?

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
Deil's user avatar
  • 193
1 vote
0 answers
225 views

How does attention for feature fusion works

I am struggling to understand how would a self-attention layer be used for features of different modalities fusion. What I understand until now is that : Every unique modality is fed into a self-...
JoongKi's user avatar
  • 11
1 vote
0 answers
313 views

How to use in built Keras ADDITIVE ATTENTION Layer for image captioning?

I have Designed an Encoder-Decoder Model for Image Captioning. Now, I want to improve my Model. So, I thought of putting an Attention Layer in my Encoder-Decoder model. But, I am struggling with how ...
Harsh Walia's user avatar
1 vote
0 answers
26 views

SAGAN - what is the correct architecture?

Hi, in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
Ilya.K.'s user avatar
  • 157
1 vote
0 answers
95 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
ihadanny's user avatar
  • 1,357
1 vote
1 answer
113 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
Deshwal's user avatar
  • 323
1 vote
2 answers
303 views

Is a dense layer required for implementing Bahdanau attention?

I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed. This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
Lucky Man's user avatar
1 vote
1 answer
64 views

Predicting point sequence in image

My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images. I am ...
arrhhh's user avatar
  • 41
1 vote
1 answer
1k views

How to add a Decoder & Attention Layer to Bidirectional Encoder with tensorflow 2.0

I am a beginner in machine learning and I'm trying to create a spelling correction model that spell checks for a small amount of vocab (approximately 1000 phrases). Currently, I am refering to the ...
Dom's user avatar
  • 11
1 vote
1 answer
61 views

Attention network without hidden state?

I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like: ...
JMRC's user avatar
  • 111
1 vote
1 answer
48 views

Attention to multiple areas of same sentence

Lets consider some sentences below: "Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too" "Can company1 buy company2? ...
Sandeep Bhutani's user avatar
1 vote
0 answers
13 views

Training a model for Single Image Super Reoslution

I'm trying to implement the Attention-based approach for SISR paper. However, during something odd happens. The MAE for the first output of the model is very small. But as the training progresses, the ...
Dhanush Giriyan's user avatar
1 vote
0 answers
249 views

How to Visualize Graph Attention

I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to ...
sajjad.riaj's user avatar
0 votes
0 answers
12 views

How can self-attention be used to combine representations from long text?

The paper "How to Fine-Tune BERT for Text Classification?" discusses using self-attention to combine the representations of a long input text that has been broken into chunks (section 5.3.1)....
suse's user avatar
  • 3
0 votes
0 answers
4 views

Intuition and technical explanation behind "3D-aware Feature Attention" in Sync Dreamer? (Multiview Generating Diffusion Model)

I am looking to understand the whole block on how the Sync Dreamer paper has constructed the "3D aware feature attention" which is outlined on page 5. To enforce consistency among multiple ...
ChaoS Adm's user avatar
  • 141
0 votes
0 answers
50 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
Reinis Mazeiks's user avatar
0 votes
0 answers
11 views

What is the prior mu in Heterogeneous Graph Transformer?

I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
Servus's user avatar
  • 101
0 votes
0 answers
36 views

Math Behind Additive Bahdanau Attention

I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism. The hidden state ...
user154214's user avatar
0 votes
0 answers
93 views

Is it a good idea to use attention in VAEs for image generation?

There are research papers and codebases on GitHub that deal with VAEs for image generation on popular datasets like CelebA, etc. While surfing through Google Scholar I found self-attention and other ...
Sir Arthur7's user avatar
0 votes
0 answers
40 views

Are there other "interactive" non-linear neural network layers besides self-attention layer?

In the self-attention layer $$ \operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$ $Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
DeLorean88's user avatar
0 votes
0 answers
39 views

Cross entropy loss starts out very low

I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For ...
Justin Goodrich's user avatar
0 votes
0 answers
35 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
168 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 1,165
0 votes
0 answers
25 views

Are Soft and Bahdanau attentions different?

I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like ...
Naveen Reddy Marthala's user avatar
0 votes
0 answers
16 views

Does this kind of attention exist?

As someone who is new to deep learning, I am only familiar with self-attention. I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
user900476's user avatar
0 votes
0 answers
811 views

Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?

I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
Aushilfsgott's user avatar
0 votes
2 answers
332 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
Jozdien's user avatar
0 votes
0 answers
226 views

How to use flatten with SeqSelfAttention

I want to use SeqSelfAttention , but in final layer the dimension need to be reduced. However, adding Flatten gives following error : ...
Sandeep Bhutani's user avatar
0 votes
0 answers
245 views

Accuracy goes low with attention layer

Below is code 1 which is not using Attention layer : ...
Sandeep Bhutani's user avatar
0 votes
0 answers
35 views

An algorithm to extract the purpose of a document

I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
Yassine's user avatar
  • 35