Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

Custom Simulator for Deep Reinforcement Learning

I am trying to develop a control method for a specific process in industry. I have a time-series of data for the process and want to develop a prediction model base on attention mechanism to estimate ...
user avatar
1 vote
0 answers
23 views

How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting

Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights. ...
user avatar
0 votes
0 answers
34 views

Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?

I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
user avatar
0 votes
0 answers
17 views

Document understanding - sentence length prediction

the subject is taken almost verbatim from this paper https://arxiv.org/pdf/2108.02923.pdf. One of the tasks , is to be able to tell, in a document, if 2 words are part of the same phrase. For e.g. if ...
user avatar
0 votes
0 answers
42 views

How to implement Early stopping in Neural Machine Translation with Attention or Transformers?

I am trying to implement early stopping to my model where I am performing Machine Translation using Seq2Seq with attention. I am mostly used to writing my own models in steps, something like this: <...
user avatar
0 votes
0 answers
39 views

Input 0 is incompatible with layer repeat_vector_40: expected ndim=2, found ndim=1

I am developing an LSTM autoencoder model for anomaly detection. I have my keras model setup as below: ...
user avatar
  • 13
0 votes
1 answer
21 views

For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq) But when using ...
user avatar
  • 3
0 votes
1 answer
105 views

Positional encoding without input embedding

Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer? In NLP models, the embedding maps a word to real numbers. ...
user avatar
0 votes
0 answers
55 views

How to extract attention weight from MultiHeadAttention layer in Keras?

How to extract attention weights from MultiHeadAttention layer in Keras? With the attention weights, I hope to plot attention heatmaps.
user avatar
2 votes
0 answers
132 views

How to use the keras.layers.AdditiveAttention correctly?

My understanding on the topic is superficial at best, so do bear with me. I have a couple questions (specifically on how to use keras.layers.AdditiveAttention) which I hope is suitable to be asked ...
user avatar
  • 143
0 votes
0 answers
42 views

keras_self_attention vs attention layer

can i get a help pleas what is the diffreent btween if i bulid attention class like this to my Bi LSTM MODEL and keras_self_attention ? from tensorflow.keras.layers import Layer from tensorflow.keras ...
user avatar
0 votes
0 answers
22 views

Why does Bahdanau Attention Have to be Causal?

Using the Bahdanau attention layer on Tensorflow for time series prediction, although conceptually it is similar to NLP applications. This is how the minimal example code for a single layer looks like....
user avatar
  • 283
1 vote
1 answer
227 views

Self Attention vs LSTM with Attention for NMT

I am trying to compare the A: Transformer-based architecture for Neural Machine Translation (NMT) from the Attention is All You Need paper, with B: an architecture based on Bi-directional LSTM's in ...
user avatar
0 votes
0 answers
32 views

Pytorch different result when using `torch.matmul` and `for-loop` to pass input through linear layers

I have been at this for about two days now. I am working on a model that takes on an input x and passes it through several linear layers, and concatenates the ...
user avatar
  • 121
0 votes
0 answers
11 views

How to reference length of input sequence inside Keras model?

Basically I am trying to implement a "Transformer-like" architecture for physiological time-series classification. One of the specific design criteria is for the model to be able to process ...
user avatar
  • 33
1 vote
1 answer
49 views

Multioutput prediction using LSTM encoder decoder with Attention

(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.) I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
user avatar
1 vote
0 answers
18 views

Can a reformer model really handle long-range dependency?

I read this article about new attention model called Reformer. Here is the main strength of this model: The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
user avatar
0 votes
0 answers
223 views

nn.embedding alternative for float numbers

I have found this pytorch code of transformers suitable for machine translation: ...
user avatar
0 votes
1 answer
16 views

Where do Q vectors come from in Attention-based Sequence-to-Sequence Transformers?

I'm taking a course on Attention-based NLP but I'm not understanding the calculation and application of Attention, based on the use of Q, K, and V vectors. My understanding is that the K and V ...
user avatar
0 votes
0 answers
16 views

Attention to get context of words

The W2V techniques define context as a window of k words around the term, and using this learn the vector representations for words in the corpus. Attention networks can help us get the important ...
user avatar
  • 37
1 vote
1 answer
127 views

How is attention different from linear MLPs?

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
user avatar
0 votes
0 answers
52 views

Empty prediction with keras Seq2Seq with attention mechanism

I have a simply seq2seq model with attention mechanism in keras. My problem is that the inference model only gives me empty prediction. However, if I remove the attention it suvessfully gives me the ...
user avatar
  • 103
3 votes
1 answer
174 views

What reccent alternatives to LSTM are there for regression problems?

I have been working for a while on a regression problem - predicting the air pollution in a city based on meteorological features (humidity, temperature, wind velocity a.o.). I have trained an LSTM ...
user avatar
  • 228
0 votes
0 answers
11 views

Rstudio Keras Attention manual calculation disagrees with the layer output

I am trying to reproduce the math inside the Keras Attention (dot-product), based on Rstudio Keras wrapper. In R I use the ouput of an LSTM layer: sequence representation (query) and last hidden state ...
user avatar
0 votes
1 answer
83 views

time series anomaly detection

I want to ask for time series anomaly detection we can apply tnn on multiple features or not? I used transformer for sentiment analysis where I have to provide a sentence and it predicts its output as ...
user avatar
  • 175
0 votes
2 answers
86 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
user avatar
0 votes
0 answers
21 views

How to prepare data for TpyTorch's 3d attn_mask argument in MultiHeadAttention

I'm currently trying to implement an Encoder-Decoder architecture for text summarization based on Transformers. Thus I need ti apply MultiHeadAttention on the Decoder site of the model. Since I want ...
user avatar
1 vote
0 answers
20 views

Which layer actually contains attention and how to map with input for classification scenario

I am writing a binary classification case and want to use Attention. The code is here - Code and model graph is shown below with attention area highlighted . I want to visualize which which word is ...
user avatar
0 votes
1 answer
207 views

Does Keras MultiHeadAttention with 1 head equals normal self attention?

Keras multihead attention if used as single head num_heads=1, then how is it different than Keras Attention ? Also, Is multihead attention by default self-...
user avatar
0 votes
0 answers
74 views

How to use flatten with SeqSelfAttention

I want to use SeqSelfAttention , but in final layer the dimension need to be reduced. However, adding Flatten gives following error : ...
user avatar
1 vote
1 answer
103 views

Why does Keras only have 3 types of attention layers?

The Keras library list only has 3 types of attentions - keras attention layers, which are : MultiHeadAttention layer Attention layer AdditiveAttention layer However, in theory there are multiple ...
user avatar
0 votes
0 answers
43 views

Accuracy goes low with attention layer

Below is code 1 which is not using Attention layer : ...
user avatar
1 vote
1 answer
109 views

why multiple attention heads learn differently

In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
user avatar
0 votes
1 answer
106 views

Attention weights - change during learning and prediction

Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values. Please ...
user avatar
3 votes
0 answers
102 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
user avatar
  • 131
0 votes
2 answers
92 views

How to encode a sentence using an attention mechanism?

Recently, I read about one of the state-of-the-art method called Attention models. This method use a Encoder-Decoder model. It can find a better encoding for each word in a sentence. But how can I ...
user avatar
0 votes
0 answers
34 views

An algorithm to extract the purpose of a document

I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
user avatar
  • 35
1 vote
0 answers
20 views

Why do we need dot product as part of the Transformer's training process?

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
user avatar
  • 183
0 votes
0 answers
108 views

Understanding difference between benefits of having multi-head attention and of the process of learning Q,K,V embeddings in a single head

From reading different papers and blogposts I got an understanding that learning embeddings with a single attention head and with multiple heads serves essentially the same purpose. In this blogpost ...
user avatar
  • 183
2 votes
1 answer
222 views

In Transformer's multi-headed attention, how attending "different representation subspaces at different positions" is achieved?

Question partially inspired by this post about the need of multi-head attention mechanism. For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(...
user avatar
  • 183
0 votes
0 answers
17 views

Computing Attention on multi-dimensional sequences?

Is it possible to compute attention/adapt existing transformer architectures (like longformer) to be used on multi-dimensional sequence input? As in, instead of a ...
user avatar
  • 157
2 votes
1 answer
356 views

What exactly is the linear layer in the transformer model?

Please see this image: There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here: Are these linear layers ...
user avatar
  • 23
1 vote
1 answer
117 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
user avatar
  • 157
2 votes
1 answer
323 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
user avatar
  • 261
1 vote
0 answers
75 views

How does attention for feature fusion works

I am struggling to understand how would a self-attention layer be used for features of different modalities fusion. What I understand until now is that : Every unique modality is fed into a self-...
user avatar
  • 11
1 vote
1 answer
53 views

Why are convolutions still used in some Transformer networks for speech enhancement?

So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...
user avatar
2 votes
1 answer
185 views

Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
user avatar
0 votes
2 answers
61 views

Intuition of "Head" in Attention models (Transformer)?

I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!
user avatar
  • 2,475
1 vote
0 answers
237 views

How to use in built Keras ADDITIVE ATTENTION Layer for image captioning?

I have Designed an Encoder-Decoder Model for Image Captioning. Now, I want to improve my Model. So, I thought of putting an Attention Layer in my Encoder-Decoder model. But, I am struggling with how ...
user avatar
1 vote
0 answers
19 views

SAGAN - what is the correct architecture?

Hi, in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
user avatar
  • 157