Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
136
questions
0
votes
0
answers
15
views
Selecting an element in a sequence with self-attention networks
I have a doubt on I should set up the following problem:
Data:
My data is a tensor with shape (N, J, F) where N is the batch size, J is the sequence length, and F is the number of features of each ...
0
votes
1
answer
18
views
How many parameters does the vanilla Transformer have?
The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
- 163
0
votes
0
answers
9
views
Training and Validation accuracy decreases after using more data
I have a binary classification project, I use a neural network with the following architecture:
The shape of the input is 64×64×4. This input was fed to a Conv2D layer with 32(5×5) filters followed by ...
0
votes
0
answers
9
views
In "Show, attend and tell", why do the attention weights get multiplied with the features to form the context vector?
The attention weights are formed through the last hidden state of the LSTM and the feature map from some kind of image encoder (in my case resnet so the features are in the form of 14x14x2048). They ...
- 101
1
vote
1
answer
41
views
Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?
By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)
Still, I got a question from reading there, hopefully, you guys can help me out
Are the ...
- 113
0
votes
0
answers
55
views
can I use tf.keras.layers.MultiHeadAttention for image classification task?
Could you please let me know whether it is possible to use tf.keras.layers.MultiHeadAttention() for the image classification task without using Vision Transformer techniqe?
0
votes
1
answer
76
views
Vision Transformer ViT Parameter count
The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021)
includes the following table:
Can someone explain how they get the parameter counts or where my calculation is ...
- 111
0
votes
0
answers
14
views
Why do transformers operate on discrete sequences?
Why do we need to discretize our input $x$ vectors in transformers? For example, we often employ VQ-VAE's to discretize images to interface with ViTs.
Surely, because attention calculation simply ...
- 197
1
vote
0
answers
24
views
Is normalization of word embeddings important?
I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
0
votes
1
answer
48
views
Is there bias in matrix multiplications for self attention
When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well.
...
- 13
0
votes
0
answers
10
views
How can i increase accuracy of my fine tuned T-5 text summarizer?
I am working on text summarization, I have fine-tuned of T-5 model with my dataset. I am using a small dataset. I have to perform with this dataset. Now I am facing two problems.
1 - Low Accuracy on ...
2
votes
1
answer
52
views
Transformer XL - understanding paper's illustration
If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where ...
- 121
1
vote
0
answers
117
views
Keras NLP TransformerDecoder MultiHeadAttention Value Error
Recently I have been working on a MIDI Music Generator using the TransformerEncoder & TransformerDecoder layers found in the Keras NLP library. There is not much info/help on these layers which is ...
- 11
1
vote
2
answers
92
views
How do transformers differ from feature selection and regular machine learning?
This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
- 91
0
votes
0
answers
22
views
Are Soft and Bahdanau attentions different?
I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like ...
0
votes
1
answer
186
views
Multi head self attention output size for batches with different sequence length
I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same ...
- 3
0
votes
0
answers
79
views
Is there an example of the implementation of the Keras Attention Layer? The example code in the website is incomplete
I have been struggling for a few weeks trying to implement an Attention layer to a bidirectional LSTM model that I've built.
Keras has three different kinds of Attention layer, and I'd like to see at ...
- 11
0
votes
1
answer
78
views
How to use strong labels in image classification?
I have a dataset where I have localized pixel-level annotations of a dataset of cancer vs non-cancer.
Which deep learning methods can I use to optimize the model to focus on the localized regions of ...
- 33
0
votes
0
answers
287
views
How to add an Attention Layer on Keras?
0
I am trying to replicate a BiLSTM model that I read on an article, but I am having a lot of trouble implementing an Attention Layer. I am not particularly concerned that it has to be the exact same ...
- 11
1
vote
1
answer
518
views
Difference Between Attention and Fully Connected Layers in Deep Learning
There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
- 726
0
votes
0
answers
15
views
Does this kind of attention exist?
As someone who is new to deep learning, I am only familiar with self-attention.
I'm designing a model. Imagine there are n data, which the $i_{th}$ data can be represented as a vector $x_i$. And the ...
- 165
0
votes
0
answers
22
views
Changing attention mechanism for average pool
I am using the TFNET model from this github for a ML project. I am trying to compare the difference between using the attention mechanism and just using plain max/average pooling. As this is not my ...
- 1
0
votes
1
answer
71
views
Transformers - Why Self Attention calculate dot product of q and k from of same word?
As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ ...
- 533
0
votes
1
answer
55
views
Visualize attention area
I wonder how people draw a network's attention area on a single input.
Such as:
Any hint is much appreciated
- 13
1
vote
1
answer
549
views
How to add the Luong Attention Mechanism into CNN?
As I write my CNN model for an image binary classification below, I'm trying to add an attention layer to this model. I read from tf.keras.layers.Attention: https://...
- 11
0
votes
0
answers
87
views
What is a in intuitive explanation of attention and self attention mechanisms?
...that goes into sufficient technical details. I could not find any great resource out there. I think an infographics in this fashion:
could come a long way, if that makes sense here.
Credit to ...
1
vote
0
answers
107
views
Could Attention_mask in T5 be a float in [0,1]?
I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as
...
- 13
2
votes
0
answers
15
views
Custom Simulator for Deep Reinforcement Learning
I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
1
vote
1
answer
612
views
How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting
Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights.
...
- 113
0
votes
0
answers
308
views
Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?
I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
0
votes
0
answers
24
views
Document understanding - sentence length prediction
the subject is taken almost verbatim from this paper https://arxiv.org/pdf/2108.02923.pdf. One of the tasks , is to be able to tell, in a document, if 2 words are part of the same phrase. For e.g. if ...
- 308
0
votes
1
answer
83
views
For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?
The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See:
Why does LSTM performs better when the source target is reversed? (Seq2seq)
But when using ...
- 3
0
votes
1
answer
513
views
Positional encoding without input embedding
Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer?
In NLP models, the embedding maps a word to real numbers. ...
- 103
2
votes
0
answers
154
views
How to use the keras.layers.AdditiveAttention correctly?
My understanding on the topic is superficial at best, so do bear with me. I have a couple questions (specifically on how to use keras.layers.AdditiveAttention) which I hope is suitable to be asked ...
- 173
1
vote
0
answers
61
views
Why does Bahdanau Attention Have to be Causal?
Using the Bahdanau attention layer on Tensorflow for time series prediction, although conceptually it is similar to NLP applications.
This is how the minimal example code for a single layer looks like....
- 303
1
vote
2
answers
1k
views
Self Attention vs LSTM with Attention for NMT
I am trying to compare the
A: Transformer-based architecture for Neural Machine Translation (NMT) from the Attention is All You Need paper, with
B: an architecture based on Bi-directional LSTM's in ...
- 11
1
vote
1
answer
140
views
Multioutput prediction using LSTM encoder decoder with Attention
(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.)
I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
1
vote
0
answers
25
views
Can a reformer model really handle long-range dependency?
I read this article about new attention model called Reformer. Here is the main strength of this model:
The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
0
votes
1
answer
30
views
Where do Q vectors come from in Attention-based Sequence-to-Sequence Transformers?
I'm taking a course on Attention-based NLP but I'm not understanding the calculation and application of Attention, based on the use of Q, K, and V vectors. My understanding is that the K and V ...
- 103
3
votes
2
answers
634
views
How is attention different from linear MLPs?
Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
3
votes
1
answer
528
views
What reccent alternatives to LSTM are there for regression problems?
I have been working for a while on a regression problem - predicting the air pollution in a city based on meteorological features (humidity, temperature, wind velocity a.o.). I have trained an LSTM ...
- 238
0
votes
1
answer
85
views
time series anomaly detection
I want to ask for time series anomaly detection we can apply tnn on multiple features or not?
I used transformer for sentiment analysis where I have to provide a sentence and it predicts its output as ...
- 171
0
votes
2
answers
166
views
Self-Attention Summation and Loss of Information
In self-attention, the attention for a word is calculated as:
$$
A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>}
$$
My question is why we sum over the ...
- 1
1
vote
0
answers
39
views
Which layer actually contains attention and how to map with input for classification scenario
I am writing a binary classification case and want to use Attention. The code is here - Code and model graph is shown below with attention area highlighted .
I want to visualize which which word is ...
- 864
1
vote
1
answer
503
views
Does Keras MultiHeadAttention with 1 head equals normal self attention?
Keras multihead attention if used as single head num_heads=1, then how is it different than Keras Attention ?
Also, Is multihead attention by default self-...
- 864
0
votes
0
answers
145
views
How to use flatten with SeqSelfAttention
I want to use SeqSelfAttention , but in final layer the dimension need to be reduced. However, adding Flatten gives following error : ...
- 864
2
votes
1
answer
192
views
Why does Keras only have 3 types of attention layers?
The Keras library list only has 3 types of attentions - keras attention layers, which are :
MultiHeadAttention layer
Attention layer
AdditiveAttention layer
However, in theory there are multiple ...
- 864
0
votes
0
answers
157
views
1
vote
1
answer
302
views
why multiple attention heads learn differently
In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
- 864
0
votes
1
answer
430
views
Attention weights - change during learning and prediction
Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values.
Please ...
- 864