Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
0
votes
0answers
7 views

How to prepare data for TpyTorch's 3d attn_mask argument in MultiHeadAttention

I'm currently trying to implement an Encoder-Decoder architecture for text summarization based on Transformers. Thus I need ti apply MultiHeadAttention on the Decoder site of the model. Since I want ...
1
vote
0answers
11 views

Which layer actually contains attention and how to map with input for classification scenario

I am writing a binary classification case and want to use Attention. The code is here - Code and model graph is shown below with attention area highlighted . I want to visualize which which word is ...
0
votes
1answer
31 views

Does Keras MultiHeadAttention with 1 head equals normal self attention?

Keras multihead attention if used as single head num_heads=1, then how is it different than Keras Attention ? Also, Is multihead attention by default self-...
0
votes
0answers
21 views

How to use flatten with SeqSelfAttention

I want to use SeqSelfAttention , but in final layer the dimension need to be reduced. However, adding Flatten gives following error : ...
1
vote
1answer
27 views

Why does Keras only have 3 types of attention layers?

The Keras library list only has 3 types of attentions - keras attention layers, which are : MultiHeadAttention layer Attention layer AdditiveAttention layer However, in theory there are multiple ...
0
votes
0answers
16 views

Accuracy goes low with attention layer

Below is code 1 which is not using Attention layer : ...
0
votes
1answer
42 views

why multiple attention heads learn differently

In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
0
votes
1answer
30 views

Attention weights - change during learning and prediction

Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values. Please ...
3
votes
0answers
29 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
0
votes
2answers
40 views

How to encode a sentence using an attention mechanism?

Recently, I read about one of the state-of-the-art method called Attention models. This method use a Encoder-Decoder model. It can find a better encoding for each word in a sentence. But how can I ...
0
votes
0answers
33 views

An algorithm to extract the purpose of a document

I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
1
vote
0answers
18 views

Why do we need dot product as part of the Transformer's training process?

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
0
votes
0answers
19 views

Understanding difference between benefits of having multi-head attention and of the process of learning Q,K,V embeddings in a single head

From reading different papers and blogposts I got an understanding that learning embeddings with a single attention head and with multiple heads serves essentially the same purpose. In this blogpost ...
1
vote
1answer
53 views

In Transformer's multi-headed attention, how attending “different representation subspaces at different positions” is achieved?

Question partially inspired by this post about the need of multi-head attention mechanism. For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(...
0
votes
0answers
13 views

Computing Attention on multi-dimensional sequences?

Is it possible to compute attention/adapt existing transformer architectures (like longformer) to be used on multi-dimensional sequence input? As in, instead of a ...
2
votes
1answer
45 views

What exactly is the linear layer in the transformer model?

Please see this image: There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here: Are these linear layers ...
1
vote
1answer
23 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
0
votes
0answers
29 views

Visualization of SeqSelfAttention layer from Keras

Can someone help me to find, explain and visualize the SeqSelfAttention layer from Keras. I found a lot of flowchart that use the figures from the "attention is all you need paper" where ...
1
vote
0answers
31 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
2
votes
0answers
25 views

How does attention for feature fusion works

I am struggling to understand how would a self-attention layer be used for features of different modalities fusion. What I understand until now is that : Every unique modality is fed into a self-...
0
votes
0answers
18 views

Output shapes of Keras AdditiveAttention Layer

Trying to use the AdditiveAttention layer in Keras. On manual implementation of the layer from tensorflow tutorial https://www....
0
votes
0answers
18 views

Feature fusion multiple modalities example

I am trying to replicate this paper. In their architecture, there is an attention-fusion subnet explained as follows: In the multimodal activity recognition problem, not all modalities are equally ...
0
votes
0answers
17 views

Feature fusion with attention layer

I have three different modalities, each of which has a shape of (n-users, m-timestep, p-features). I want to do feature level fusion, by adding an attention layer that would compute the scores of ...
1
vote
1answer
40 views

Why are convolutions still used in some Transformer networks for speech enhancement?

So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...
2
votes
1answer
80 views

Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
0
votes
0answers
30 views

How does multi-head attention on “multiple attention axes” works?

I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a ...
0
votes
2answers
35 views

Intuition of “Head” in Attention models (Transformer)?

I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!
1
vote
0answers
109 views

How to use in built Keras ADDITIVE ATTENTION Layer for image captioning?

I have Designed an Encoder-Decoder Model for Image Captioning. Now, I want to improve my Model. So, I thought of putting an Attention Layer in my Encoder-Decoder model. But, I am struggling with how ...
1
vote
0answers
19 views

SAGAN - what is the correct architecture?

Hi, in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
0
votes
0answers
105 views

Why do relative positional embeddings instead of absolute positional encoding improve the Transformer?

I've been researching the Music Transformer, the paper for which introduced an efficient algorithm to compute Relative Positional Embeddings in a Transformer. I know that Relative Positional ...
0
votes
1answer
21 views

There could be a problem with the linear layer after the attention inside a transformer?

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...
0
votes
1answer
749 views

Class token in ViT and BERT

I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT. To the best of my understanding this token is used to gather knowledge of the entire class, ...
1
vote
1answer
73 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
0
votes
1answer
55 views

Attention in NER

I am building a named entity recognition model and it is having BERT+BiLSTM+CRF in it. Now I am planning to introduce an attention layer. My question is - what type of attention I should use here and ...
0
votes
0answers
14 views

Language Model with Attention not learning

Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7% The way input data is pipelined is by ...
2
votes
1answer
90 views

How do attention mechanism in CNN for images?

I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the ...
0
votes
1answer
28 views

Transformer architecture question

I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/. The first attention ...
2
votes
1answer
37 views

Nutritional image classification task

I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar ...
0
votes
1answer
105 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
1
vote
0answers
28 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
7
votes
2answers
768 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
1answer
56 views

Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?

I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. ...
0
votes
0answers
23 views

Basic of the attention mechanism

Background Having gone through articles. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Attention in Neural Networks Visualizing A Neural Machine Translation Model (...
0
votes
1answer
50 views

Predict customer behaviour with Transformer(attention is all you need)

Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
3
votes
1answer
612 views

Attention for time-series in neural networks

Neural networks in many domains (audio, video, image text/NLP) can achieve great results. In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - ...
1
vote
2answers
183 views

Role of decoder in Transformer?

I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
119 views

Why this TensorFlow Transformer model has Linear output instead of Softmax?

I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation. I am quite surprised that when the Transformer is created, their final output is a Dense ...
0
votes
0answers
11 views

Using Transcoder Model for language to language conversion

I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
1
vote
1answer
32 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...